Databricks Data Engineer

Overview

As a Senior Data Engineer, you will design scalable, real-time data pipelines and platforms that power analytics, ML, and operational systems for a leading online retailer. You’ll collaborate with cross-functional teams to ensure high-quality, compliant data at scale while applying modern data tooling. This hands-on role blends software craftsmanship with data architecture to enable data-driven decisions at scale. You will shape data governance, observability, and streaming capabilities to support personalised customer experiences.

Responsibilities

• Design and implement high-throughput data streaming solutions using Kafka, Flink, or Confluent
• Build and maintain scalable backend systems in Python or Scala with clean code and tests
• Develop data governance, privacy, and quality monitoring tools to ensure compliance
• Create robust data workflows and automation within Airflow, Databricks, and other big data ecosystems
• Implement data observability and cataloguing tools (Monte Carlo, Atlan, DataHub)
• Collaborate with ML engineers, analysts, and analytics engineers to meet data needs
• Promote testing, peer reviews, and automation-first engineering culture

Key requirements

• Strong experience in streaming technologies (Kafka, Flink, or Confluent)
• Advanced proficiency in Python or Scala with solid software engineering fundamentals
• Proven ability to design, deploy, and scale production-grade data platforms and backend systems
• Familiarity with data governance, privacy compliance, and automated data quality checks
• Hands-on experience with big data tools (Airflow, Databricks) and data observability platforms
• Collaborative mindset with cross-functional teams including ML and analytics
• Curiosity and enthusiasm for continuous learning

• Collaborative mindset
• Curiosity and continuous learning
• Team-oriented communication
• Kafka
• Flink
• Confluent

…