AI Evaluation & Quality Engineer

WeQuelStockholmJob.bopublished 11/20/2026
Must-have:PythonQA/TestCI/CDAIHybrid

AI Evaluation & Quality Engineer

WeQuel is looking for an AI Evaluation & Quality Engineer for an assignment with one of our clients. You will make sure agentic workflows are measured honestly before and after they reach production – so the organisation knows what works, what does not, and why.

About the assignment

  • Workload: 100%
  • Location: Hybrid
  • Language: English

Your responsibilities

  • Design evaluation strategies and quality metrics for LLM agents (accuracy, task success, safety, cost, latency)
  • Build and maintain evaluation datasets, test cases and automated regression tests
  • Set up LLM-as-judge and human review processes where needed
  • Run red-teaming and robustness tests, including prompt injection scenarios
  • Monitor agent quality in production and feed findings back to development teams
  • Report results clearly to product owners and stakeholders

Requirements

  • At least 3 years of experience in test automation, QA engineering or data science
  • Hands-on experience evaluating LLM or ML solutions
  • Strong Python skills
  • Understanding of statistics and experiment design
  • Fluent English, written and spoken

Meritorious

  • Experience with evaluation and observability tools (e.g. Langfuse, LangSmith, Ragas, DeepEval, promptfoo)
  • Experience with CI/CD pipelines for ML or LLM applications
  • Knowledge of AI safety and the EU AI Act
  • Background in industrial or manufacturing environments
  • Swedish

Who you are

You are analytical, sceptical in a constructive way and thorough. You enjoy finding where things break, explain results in plain language, and help teams make better decisions based on evidence.

Applications are reviewed continuously – apply today!