AI Engineering Lead, Evals and Systems Integration

Overview

In this role you will own the evaluation framework and cross-team system integration for Bain’s Growth AI Performance System. You’ll shape how we measure quality across data, agentic cores, and product modules, ensuring reliable, deploy-ready AI solutions for large clients. You’ll work closely with data, platform, and client teams to translate feedback into tangible improvements, balancing rigorous evaluation with practical delivery. This position offers startup-like ownership within a global firm and an opportunity to impact real client outcomes through responsible, scalable AI.

Pay / Benefits

• base salary and discretionary performance bonus
• 401(k) with employer contribution
• paid time off including parental leave and holidays
• fully vested 401(k) contribution
• medical, dental and vision premiums covered
• life and long-term disability insurance

Responsibilities

• Own and evolve the end-to-end evaluation framework across the full platform (data ontology, Agentic Core, product modules)
• Design and maintain evaluation datasets and regression suites for cross-stack quality measurement
• Define deployment-readiness standards at each layer and enforce them before live deployments
• Create experiments with statistical design to detect quality issues and root causes
• Build CI/CD-integrated automated evaluation pipelines with continuous quality gates
• Evaluate integration points between knowledge graph, agentic workflows, and API contracts
• Coordinate evaluation work across distributed teams and communicate findings to leadership
• Engage with client-facing teams post-deployments to translate feedback into improvements
• Stay current with evaluation methodologies and responsible AI practices

Key requirements

• 8+ years in AI/ML engineering or applied AI with hands-on build responsibilities
• Proven track record of building and owning production evaluation frameworks
• Experience evaluating system-level integrations across data pipelines, APIs, and agentic workflows
• Strong Python skills; experience implementing automated evaluation pipelines and CI/CD integration
• Statistical reasoning and experimental design to measure system quality
• Familiarity with knowledge graphs, ontology evaluation, entity resolution, and semantic data concepts
• Experience with human-in-the-loop annotation and feedback systems for LLM/agentic apps
• Understanding of security, audit, and compliance for evaluation infrastructure
• Native AI fluency with AI-assisted coding tools and ongoing methodology development
• Commitment to responsible AI, including bias/fairness assessment and AI safety principles

• strong cross-team collaboration
• clear stakeholder communication
• problem solving under uncertainty
• evaluation framework design for LLM/agentic systems
• systems integration testing across modules and data contracts
• regression test suites and scoring rubrics

…