Lead Databricks Engineer

Overview

In this role, you will design and lead enterprise-scale Azure Databricks data platforms to enable advanced analytics and real-time processing. You will shape platform architecture, governance, and cost optimization to deliver high-quality solutions for critical business needs in a data-driven environment. You’ll work with cross-functional teams to translate requirements into robust technical outcomes and drive production-ready platforms. This position offers an opportunity to influence platform strategy and governance at scale, within a hybrid-working setup in London.

Responsibilities

• Lead architecture design and implementation of scalable solutions on Azure Databricks
• Define and enforce platform standards, ensuring data governance and security policy compliance
• Build and optimize Databricks streaming workloads (Structured Streaming, Delta Live Tables)
• Collaborate with cross-functional teams to translate requirements into robust solutions
• Drive platform performance tuning, cost optimization, and monitoring for cloud workloads
• Provide leadership across the engineering lifecycle, including delivery and production support
• Mentor and guide developers, establish best practices for data engineering on Azure and Databricks
• Participate in technical decision-making and design reviews for scalability and maintainability
• Implement observability for critical pipelines and maintain QA standards
• Engage with stakeholders to align on strategic platform initiatives

Key requirements

• 8+ years in data engineering, including 3+ years in a tech lead capacity
• Strong hands-on Databricks experience for enterprise-scale workloads
• Proficiency in PySpark, Spark Structured Streaming, and Delta Lake
• Python programming and SQL performance optimization experience
• Deep understanding of Azure Data Platform services and cloud-native patterns
• Experience implementing data governance, quality management and observability frameworks
• Ability to manage complex data pipelines for batch and streaming use cases
• Excellent communication and leadership skills for distributed teams and senior stakeholders
• Knowledge of Delta Live Tables (DLT) and advanced Databricks workflows
• Familiarity with CI/CD for data engineering using GitHub or Azure DevOps
• Experience designing infrastructure-as-code solutions for Azure
• Understanding of data mesh or lakehouse principles for large-scale architectures
• Background in financial trading or capital markets data domains

• strong communication
• leadership
• stakeholder engagement
• Databricks on Azure
• PySpark
• Spark Structured Streaming

…