Senior QA Automation Engineer – AI

Overview

In this role you will own quality for GenAI/Agentic features, designing testing strategies and metrics for LLM-powered applications. You’ll build end-to-end automated tests, evaluate model behavior, and establish multi-dimensional quality criteria. You will work closely with AI and engineering teams to ensure reliability, safety, and performance in production. This is a chance to shape how AI features are tested, trusted, and adopted at scale. Join a culture that values innovation, impact, and cross-functional collaboration to drive customer success.

Responsibilities

• Design test plans and evaluation frameworks for LLM-powered applications
• Develop automated test suites in Python covering evaluations, API, UI, and performance
• Create automated prompt testing and regression suites to monitor model behavior
• Establish criteria to measure GenAI quality across accuracy, relevance, safety, consistency, and latency
• Perform red-teaming to identify biases, hallucinations, and security issues
• Build internal testing tools and frameworks for SE teams to validate GenAI implementations
• Deploy real-time monitoring and alerting to detect production quality degradation
• Own overall testing strategy and maintain high-fidelity test data and QA best practices

Key requirements

• Deep understanding of GenAI technologies (LLMs, prompt engineering)
• Strong Python coding experience
• Experience with test automation (Python, JavaScript, Selenium, Pytest)
• Background in functional, integration, regression, performance, and security testing
• Ability to design test cases for non-deterministic systems
• Experience with API testing tools (Postman, REST Assured) and backend testing
• CI/CD workflows and automated testing integration

• analytical thinking
• problem-solving
• attention to detail
• Python
• JavaScript
• Selenium

…