AI Evaluation & Quality Engineer
Must-have:PythonQA/TestCI/CDAIHybrid
AI Evaluation & Quality Engineer
WeQuel is looking for an AI Evaluation & Quality Engineer for an assignment with one of our clients. You will make sure agentic workflows are measured honestly before and after they reach production – so the organisation knows what works, what does not, and why.
About the assignment
- Workload: 100%
- Location: Hybrid
- Language: English
Your responsibilities
- Design evaluation strategies and quality metrics for LLM agents (accuracy, task success, safety, cost, latency)
- Build and maintain evaluation datasets, test cases and automated regression tests
- Set up LLM-as-judge and human review processes where needed
- Run red-teaming and robustness tests, including prompt injection scenarios
- Monitor agent quality in production and feed findings back to development teams
- Report results clearly to product owners and stakeholders
Requirements
- At least 3 years of experience in test automation, QA engineering or data science
- Hands-on experience evaluating LLM or ML solutions
- Strong Python skills
- Understanding of statistics and experiment design
- Fluent English, written and spoken
Meritorious
- Experience with evaluation and observability tools (e.g. Langfuse, LangSmith, Ragas, DeepEval, promptfoo)
- Experience with CI/CD pipelines for ML or LLM applications
- Knowledge of AI safety and the EU AI Act
- Background in industrial or manufacturing environments
- Swedish
Who you are
You are analytical, sceptical in a constructive way and thorough. You enjoy finding where things break, explain results in plain language, and help teams make better decisions based on evidence.
Applications are reviewed continuously – apply today!