Lead Data & AI Engineer

Overview

As Lead Data & AI Engineer, you will design and deliver data-driven, AI-enabled capabilities on cloud platforms in a hybrid UK setting. You’ll build scalable data pipelines and governance to support analytics and intelligent applications, while ensuring reliability, performance and security for enterprise workloads. You will work with cross-functional teams to operationalize ML solutions, integrate GenAI-assisted methods, and drive data quality across platforms. This role offers the chance to shape data architecture and ML workflow practices at scale, with a focus on efficiency and governance.

Responsibilities

• Design and implement cloud-native data architectures
• Build large-scale ETL/ELT workflows for heterogeneous datasets
• Create data models optimized for AI/ML pipelines and analytics
• Develop streaming and batch pipelines using Azure Data Factory and Databricks
• Operationalize ML solutions with feature stores, model registries, and inference endpoints
• Collaborate with data scientists to deploy and monitor AI models via enterprise MLOps
• Integrate GenAI-assisted development into data workflows for automation
• Ensure data governance, lineage, cataloging and quality across platforms
• Optimize performance, control compute costs, and enable observability for critical workloads
• Mentor engineering teams and contribute to best practices in data and AI engineering

Key requirements

• Minimum 8+ years in data engineering with architecture and leadership experience
• Advanced SQL proficiency for large-scale performance tuning
• Strong Python programming skills with data engineering workflow experience
• Expertise in PySpark for distributed processing
• Hands-on Databricks experience, including Delta Lake and optimization
• Knowledge of Azure Data Factory for ETL/ELT orchestration
• Familiarity with AI/ML pipeline development and production model integration
• Experience with Gen AI-assisted development workflows for data tasks
• Understanding of CI/CD for data and ML pipelines (GitHub Actions, Azure DevOps)
• Ability to manage enterprise-scale solutions across global, compliant environments
• Prompt engineering knowledge and experience building RAG workflows
• Knowledge of Microsoft Foundry platforms
• Version control and CI/CD experience with GitHub
• Understanding of ETL/ELT optimization patterns beyond Azure
• Knowledge of data mesh or lakehouse concepts
• Hands-on exposure to dbt, MkDocs, and similar developer tools

• Leading and mentoring teams
• Cross-functional collaboration
• Communication and stakeholder management
• Python
• SQL
• PySpark

…