Senior Data Engineer

Overview

Senior Data Engineer contract in London leading technical direction across Azure-based data platform. You’ll improve patterns, ensure quality and performance, and drive DevOps enhancements while collaborating with data, analytics and platform teams. The role focuses on optimizing a large data estate, establishing standards, and enabling scalable data pipelines. You’ll work autonomously to challenge approaches and deliver rapid impact in a global organization.

Responsibilities

• Design and improve engineering patterns across Azure Databricks and Azure Data Factory
• Drive performance optimization of data pipelines (caching, indexing, partitioning, cost)
• Review code and act as a technical quality gate
• Improve DevOps practices (CI/CD, environment promotion, branching, release management)
• Identify opportunities to automate repetitive engineering tasks
• Work with advanced SQL within established patterns
• Support data quality, governance and orchestration
• Collaborate with data engineering, analytics, data science and platform teams
• Establish consistent engineering standards across the service

Key requirements

• Azure Databricks
• Azure Data Factory
• Spark / PySpark
• Advanced SQL and performance optimisation
• CI/CD and Azure DevOps
• Delta Lake and medallion architecture
• Unity Catalog
• Data governance and orchestration
• API-based data ingestion
• Experience with high-volume operational, IoT or telemetry data is highly valuable

• Autonomous and self-driven
• Collaborative cross-functional teamwork
• Strong problem-solving and critical-thinking
• Azure Databricks
• Azure Data Factory
• Spark / PySpark

…