Lead DE (Insurance) – London

Overview

In this role, you will drive data engineering for a high-impact insurance data transformation programme. You will work across modern cloud data platforms and large-scale environments to deliver robust ETL/ELT pipelines, data integration, and governance. You’ll lead technical initiatives, optimise Databricks performance and cost, and ensure data quality and compliance within an agile, DevOps-enabled team. This is a scale-focused opportunity to shape data architecture and enable data-driven decision making in insurance. Hybrid work and clear progression make this a compelling place to apply your Azure Databricks expertise and leadership.

Responsibilities

• Design and implement ETL/ELT pipelines and data integrations on cloud platforms
• Lead technical initiatives and collaboration within an agile team
• Ensure data quality, governance, metadata and lineage across data environments
• Optimise Databricks performance, security and cost management
• Work on data modelling (Dimensional, ODS, Data Vault) and data warehousing at enterprise scale
• Collaborate with cross-functional teams, including stakeholders in policy, claims and regulatory data
• Support CI/CD processes using Git, Azure DevOps and related tools
• Contribute to data architecture decisions and platform improvements

Key requirements

• Insurance domain experience essential (policy, claims, regulatory data)
• Technical leadership experience
• Strong Azure Databricks, Azure Data Factory, Synapse & ADLS
• Advanced PySpark / SparkSQL & SQL
• ETL/ELT, data integration & pipeline orchestration
• Strong data modelling (Dimensional, ODS & Data Vault)
• Data warehousing & large-scale distributed data processing
• Data quality, EDM, governance, metadata & lineage
• Git, CI/CD, Azure DevOps & Agile
• Databricks performance, cost & security optimisation
• Exposure to AWS/GCP desirable

• technical leadership
• stakeholder collaboration
• agile mindset
• Azure Databricks
• Azure Data Factory
• Azure Synapse

…