Sr GenAI Quality & Solution Analyst (Banking, 1-year renewable contract)

Your tasks

• Design and execute end-to-end test strategies covering user interfaces, backend APIs, integrations, data flows, and agentic AI behaviour.
• Validate GenAI applications for task completion, response accuracy, relevance, grounding, consistency, citation quality, and safe failure handling.

• Test agentic workflows, including tool selection and execution, multi-turn context, state transitions, retries, timeouts, approval checkpoints, handoffs, and recovery from failures.
• Perform functional, integration, regression, exploratory, negative, resilience, and basic performance testing across application layers.

• Develop and maintain API tests covering authentication, authorisation, validation, error handling, idempotency, rate limits, and downstream dependencies.
• Automate high-value test scenarios to improve testing efficiency, reduce manual effort, and minimise production risks.

• Analyse logs, traces, request and response payloads, network calls, and database records to identify defects and determine root causes.
• Establish reusable test scenarios, test data, traceability, and evidence to support enterprise release governance and quality reporting.

• Provide evidence-based release recommendations, highlighting known limitations, residual risks, and areas requiring further monitoring.
• Collaborate with product owners, business users, architects, engineers, security teams, and GenAI specialists to define business needs, user journeys, and expected outcomes.

• Translate business requirements into functional requirements, end-to-end solution flows, interface behaviours, business rules, acceptance criteria, and non-functional requirements.
• Analyse solution options and recommend appropriate approaches involving GenAI, retrieval-augmented generation (RAG), workflow orchestration, deterministic application logic, and human approval.

• Map interactions across user interfaces, APIs, language models, prompts, retrieval components, enterprise data sources, agent tools, and downstream systems.
• Identify design gaps, integration assumptions, ownership issues, security concerns, failure scenarios, and operational risks before development begins.

• Facilitate solution design reviews and document findings through process flows, sequence diagrams, interface specifications, decision tables, and solution documentation.
• Define measurable success criteria covering business outcomes, user experience, functional correctness, response quality, latency, reliability, and safe failure.

• Maintain traceability from business requirements through solution design, implementation, test scenarios, and release evidence.

Requirements

Key Requirements

• 5–8 years of hands-on experience in software quality engineering, test engineering, or a similar role involving complex applications.
• Strong experience testing web applications, backend services, REST APIs, and enterprise integrations.

• Hands-on experience with API testing and automation tools such as Postman, REST Assured, pytest, Playwright, Cypress, Selenium, or equivalent.
• Working knowledge of Java, Python, JavaScript, or TypeScript to develop, review, and troubleshoot test automation.

• Strong test analysis skills, including requirements review, risk assessment, boundary analysis, negative testing, and traceability.
• Experience testing distributed systems and integrations, including asynchronous processing, queues, batch jobs, and downstream dependencies.

• Ability to investigate application issues using logs, traces, network calls, payloads, and database records.
• Familiarity with Git, pull requests, CI/CD pipelines, test reporting, and defect management tools.

• Understanding of security and privacy testing fundamentals, including access control, sensitive data handling, input validation, and auditability.
• Experience analysing complex applications across business processes, user journeys, APIs, data flows, and system integrations.

• Ability to facilitate requirements discussions and translate ambiguous busines needs into clear functional requirements, acceptance criteria, and solution behaviours.
• Experience producing analysis artefacts such as process flows, sequence diagrams, context diagrams, interface specifications, decision tables, and user stories.

• Understanding of application architecture concepts, including synchronous and asynchronous integrations, event flows, authentication, authorisation, system boundaries, and failure handling.
• Ability to assess solution alternatives and explain trade-offs involving quality, cost, performance, security, operational support, and delivery complexity.

• Practical understanding of LLM-based applications, RAG, prompts, context windows, embeddings, and tool-using agents.
• Ability to evaluate probabilistic AI systems using evaluation datasets, repeated test runs, quality thresholds, and evidence-based acceptance criteria rather than relying solely on exact text matching.

• Experience validating grounded responses, citations, retrieval quality, hallucination risks, prompt injection resistance, and safe handling of restricted or unsupported requests.
• Understanding of agent planning and execution, tool calls, memory and state management, human approval checkpoints, fallback behaviour, and terminationconditions.

• Familiarity with observability and evaluation tools such as Langfuse, LangSmith, OpenTelemetry, Elastic, Splunk, or equivalent.
• Strong analytical, communication, and stakeholder management skills, with the confidence to challenge unclear requirements, weak designs, and premature release decisions.

• Ability to work effectively in a fast-paced environment where requirements and GenAI behaviour continue to evolve.

Nice to Have

• Experience in banking, financial services, or other regulated enterprise environments.
• Exposure to contract testing, service virtualisation, synthetic monitoring, or performance testing.

• Familiarity with Kubernetes, OpenShift, AWS-hosted services, or containerised deployments.
• Accessibility testing experience and familiarity with WCAG-based checks.

• Experience developing quality dashboards covering release risks, defect trends, flaky tests, and testing cycle time.
• Exposure to red teaming or adversarial testing of GenAI applications.

How to apply

If you or someone you know is interested, please send the CV directly to quynh.nguyen@evolutionjobs.sg (most preferred, as I may overlook some CVs due to the high volume).

Other

Please note that visa sponsorship is not available at this time.