Senior Data Engineer : Databricks & Delta

Overview

In this role you will design and implement scalable data pipelines and enable real-time and batch analytics for a leading energy trading company. You will work with cross-functional teams to build robust data solutions that support decision-making and reporting at scale. The position focuses on Databricks, Delta Lake, streaming architectures, and SAP data ingestion, offering an opportunity to impact data-driven decisions across the organization. This contract role combines hands-on engineering with shaping data platforms to support energy trading operations.

Responsibilities

• Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)
• Develop and manage streaming data solutions for real-time analytics
• Ingest data from SAP systems using Qlik Replicate and ADF to ensure data quality
• Perform data modelling and optimize data structures for analytics and reporting
• Collaborate with cross-functional teams to understand data requirements and deliver robust solutions
• Monitor, troubleshoot, and enhance data workflows for performance and reliability
• Maintain technical documentation and contribute to data engineering best practices

Key requirements

• Strong experience with Databricks- DLT, Delta Lake concepts, UC governance
• Solid understanding of streaming technologies (e.g., Spark Structured Streaming, Autoloader)
• Programming skills in Python, SQL, or Scala
• Proficiency in data modelling, ETL/ELT processes, and data architecture
• Experience in MS Copilot, AI Foundry, Databricks (MosiacAI, MLflow, Agentbricks, Genie)
• Strong analytical background with problem-solving skills
• Performance tuning concepts like watermarking, late data handling, parallelism & checkpointing
• Hands-on expertise in ADF, and Qlik Replicate for data ingestion and replication
• Experience working in Azure cloud environments

• collaboration with cross-functional teams
• problem-solving and analytical thinking
• attention to data quality and governance
• Databricks (DLT, Delta Lake)
• Streaming technologies (Spark Structured Streaming, Autoloader)
• Python, SQL, or Scala

…