Senior AI Agent Testing and QA Consultant - M/F

Devoteam Nantes, with its 160 employees, is a key player in digital transformation in the Nantes region. Our team of expert consultants supports companies in achieving their most ambitious technological projects, relying on our expertise in cloud & DevOps, cybersecurity, data / IA, project management, and agile development. 🚀 Devoteam Creative Tech is the team specialized in designing high-performance and sustainable digital products and services with high added value. Our goal is to simplify users' daily lives and generate tangible value for our key accounts, across various sectors (banking, insurance, e-commerce, energy, industry, market finance, luxury, and retail). We are 1,100 creative technologists, including 800 experts in France, distributed across 5 expertise tribes.

As part of a strategic project, our client is developing an application intended for its business teams. The application will integrate 10 to 15 IA agents to enrich usage: automatic meeting preparation, minutes, planning, Q&A on procedures and products, email writing, etc. IA prototypes already exist and must now be industrialized. We are looking for a Senior IA QA Consultant capable of autonomously taking responsibility for project testing, in close connection with Product Owners, developers, IA/Data teams, and business units. 💪 Your missions: Define the overall test strategy for the project.

Build the specific evaluation and testing strategy for IA agents.

Write test scenarios and cases.

Perform testing of IHMs, API, and IA agents.

Specifically test the relevance and robustness of IA responses: consistency, hallucinations, reproducibility, performance, edge cases…

Create or complete the necessary test datasets.

Identify, document, and track anomalies.

Lead and coordinate business testing.

Work daily with PO, DEV, IA, Data, and business teams.

🎯 Your assets to join us: Solid experience in QA / functional and technical testing.

Significant experience in API testing.

Mastery of JIRA / Xray.

Capable of building an end-to-end test strategy.

Comfortable in an Agile environment with many stakeholders.

Capable of challenging teams and structuring testing.

Concrete experience in testing / evaluation of GenAI, LLM, or IA agent solutions is mandatory.

We will prioritize profiles that have already worked on topics such as: definition of LLM response evaluation criteria;

hallucinations and response relevance;

RAG / IA agent testing;

datasets / golden datasets;

robustness and edge cases;

comparison and qualification of IA responses.

Technical environment: Front: ReactJS

Back: Java

API

Test management / anomalies: JIRA, Xray

API Tests: Bruno

Automated test tools already present

IA / LLM technologies: technologies are currently being decided

🙋‍♂️ Why join us? We are an AI-Driven Company, partner to the largest technological partners (Google, AWS, Microsoft, ServiceNow,…) and composed of a large collective of experts (50 expertise tribes). With 75% of our employees certified, including 100% on GenAI, Devoteam establishes itself as a Learning Company, where continuous learning is at the heart of our DNA. 🤝 How does the recruitment process work at Devoteam? It consists of 2 to 3 interviews:

• Talent Acquisition Interview: the objective is to review your skills, your level of English, and to validate your motivations

• Tech & Business Interview: this interview aims to deepen your technical skills and verify their suitability with our needs during an exchange with a business expert

• Leadership Interview: it allows for the evaluation of your potential, your ambitions, and to consider your evolution within Devoteam.

We prioritize at least one in-person interview. A reference check is requested and, depending on your profile, tests (technical, English, personality...) may be sent to you.

If your application is selected, we will send you a proposal presenting the hiring conditions. Upon acceptance, the employment contract is formalized. The Devoteam Group works for equal opportunity, for the merit-based promotion of its female and male employees, and actively fights against all forms of discrimination. We are convinced that diversity contributes to the creativity, dynamism, and excellence of our organization. Each application is therefore considered independently of any discriminatory criterion.