Lead Data Engineer

Overview

As a Senior Data Engineer at Howden, you will lead the design and delivery of scalable, metadata-driven data solutions using Databricks and Azure. You will own metadata ingestion and orchestration frameworks, shaping data pipelines to enable business insights and operational efficiencies. You’ll drive best practices, CI/CD for data, and mentor junior engineers while collaborating with stakeholders across disciplines. This role places you at the heart of a modern Azure-based data ecosystem, contributing to Howden’s data-driven transformation and cross-functional impact.

Pay / Benefits

• flexible hours
• hybrid working
• reasonable adjustments
• diversity and inclusion
• career development
• global team culture

Responsibilities

• Lead metadata ingestion frameworks and orchestration processes
• Design and implement scalable, reusable metadata-driven data services
• Architect and maintain Databricks/Azure-based data solutions (Autoloader, Spark Declarative Pipelines, ADF, ADLS, Azure Functions)
• Establish and enforce data engineering, CI/CD, and DevOps best practices
• Collaborate with stakeholders, data scientists and analysts to translate data needs into solutions
• Optimize pipelines for performance, reliability, and scalability
• Lead data governance and security best practices
• Mentor junior engineers and promote continuous improvement

Key requirements

• 8+ years in data engineering focusing on Databricks (SQL & PySpark)
• Extensive Azure data services experience (ADF, ADLS, Azure Functions)
• Track record of metadata-driven pipeline design
• Expertise in orchestration and data workflow automation (Airflow, DBT)
• Strong CI/CD practices for data engineering
• Infrastructure as code experience (Terraform, Bicep, or ARM templates) preferred
• Solid software engineering standards and design discussions
• Strong stakeholder management and ability to translate requirements into data solutions
• Knowledge of data governance, security, and compliance

• leadership
• communication
• collaboration
• Databricks (SQL & PySpark)
• Azure data services (ADF, ADLS, Azure Functions)
• metadata ingestion and governance

…