Senior QA Automation Engineer / SDET - GenAI & LLM Testing - Lorien Impellam

Senior QA Automation Engineer / SDET - GenAI & LLM Testing 6 Month Contract Banking Client Remote / Hybrid (UK) Competitive Day Rate We're supporting a leading banking client on a large-scale AI transformation programme and are looking for an experienced Senior QA Automation Engineer / SDET with a strong background in GenAI and LLM testing . This role will focus on defining and implementing automated testing frameworks for AI-driven products, ensuring the quality, reliability and safety of LLM-powered solutions deployed within a highly regulated banking environment. Key Responsibilities Design and develop scalable automated test frameworks using Python. Create test strategies from architecture and solution design documentation. Test RESTful APIs and microservice-based applications. Define and execute testing approaches for event-driven architectures and Kafka-based systems. Validate GenAI and LLM solutions through functional, non-functional and behavioural testing. Develop automated prompt testing and evaluation frameworks. Verify AI guardrails, safety controls and compliance requirements. Implement validation approaches for non-deterministic LLM outputs. Build and maintain contract tests using Pact Flow. Collaborate closely with Engineering, Architecture, Data Science and Product teams. Support CI/CD pipelines and quality engineering best practices. Required Skills & Experience Core Automation Strong Python 3.x development skills with experience building testing frameworks from the ground up. Extensive experience with Pytest. Strong API testing experience. Experience creating test strategies and quality approaches from architectural diagrams and solution designs. Cloud & Platform Working knowledge of AWS services including: Lambda Bedrock S3 EKS Event-Driven Systems Experience testing Kafka-based architectures. Strong understanding of asynchronous messaging and event-driven systems. GenAI / LLM Testing (Essential) Proven experience testing LLM and GenAI applications. Prompt testing and prompt evaluation. Non-deterministic output validation. AI guardrail verification and safety testing. Hallucination and response quality assessment. Experience establishing quality standards for AI solutions. LLM Evaluation Tooling Experience with one or more of: DeepEval Promptfoo RAGAS LangSmith Similar LLM evaluation frameworks Contract Testing Hands-on experience with Pact Flow. Consumer-driven contract testing within distributed systems. Desirable Banking or Financial Services experience. Experience working in highly regulated environments. Exposure to RAG architectures. Knowledge of LLM observability and monitoring. Kubernetes experience. CI/CD pipeline integration. Guidant, Carbon60, Lorien & SRG - The Impellam Group Portfolio are acting as an Employment Business in relation to this vacancy.