SDET – AI

Overview

In this role you will design and implement automated QA strategies for GenAI features, ensuring reliable, accurate, and trustworthy AI-powered workflows. You will work across evaluations, APIs, UIs, and performance tests to validate non-deterministic AI systems. You’ll shape evaluation frameworks for GenAI metrics like accuracy and safety, and help scale testing within a cross-functional AI-focused team. This is a hands-on, impact-driven position at a leading AI-infused planning platform that serves Fortune 50 customers. You will translate complex AI behavior into robust test coverage and actionable insights.

Responsibilities

• Develop and execute end-to-end automated test coverage across evaluations, APIs, UIs, and performance tests using Python
• Implement testing strategies for GenAI features, including conversational AI and agentic systems
• Maintain automated test suites for prompt testing and regression tests to detect model behavior changes
• Support evaluation frameworks measuring GenAI performance across accuracy, relevance, safety, consistency, and latency
• Curate high-quality test datasets and golden examples representing diverse user scenarios
• Assist in adversarial testing to identify hallucinations or biases and support production alerting integration
• Utilise internal tools and testing frameworks to help developers validate GenAI code and assist with UAT
• Document testing procedures and known issues for automated coverage workstream

Key requirements

• Solid QA or test engineering experience with exposure to AI/ML systems
• Hands-on Python coding experience
• Knowledge of GenAI technologies including LLMs, prompt engineering, and AI application patterns
• Experience with test automation frameworks and scripting (Python, Pytest, API, UI Testing)
• Understanding of functional, integration, regression, performance, and security testing
• Ability to execute test cases for non-deterministic systems
• Experience with API testing tools (Postman, REST Assured) and backend testing
• Familiarity with CI/CD pipelines and automated testing integration

• Python
• Pytest
• API testing
• UI testing
• Postman
• REST Assured

…