Business Consultant – Databricks

Overview

As a Business Consultant – Databricks in London, you will design and optimize scalable Databricks pipelines, ensuring efficient performance and cost control across multi-cloud deployments. You will work with data scientists and platform teams to improve Delta Lake queries and build automated monitoring. This role focuses on tuning, orchestration, and best practices to drive reliable, high-throughput data engineering workflows. You’ll operate in a dynamic, client-facing environment shaping data engineering outcomes and infrastructure engineering practices.

Responsibilities

• Design and optimize scalable Databricks pipelines (ETL, streaming, ML workflows)
• Tune cluster and job performance (sizing, caching, partitioning, shuffle)
• Monitor Spark metrics, analyze bottlenecks in throughput or latency
• Implement autoscaling and job consolidation for cost efficiency
• Ensure efficient scheduling with Databricks Workflows
• Collaborate with data scientists and platform teams to improve Delta Lake performance
• Develop automated monitoring and alerts via Databricks REST APIs, Prometheus, or Azure Monitor
• Maintain best practices for data engineering performance, testing, and documentation

Key requirements

• Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake
• Deep understanding of Spark execution plans, shuffle optimization, and partitioning strategies
• Experience tuning job configurations (memory, parallelism, caching, adaptive query execution)
• Familiarity with orchestration tools: Databricks Workflows, Airflow, Azure Data Factory
• Proficiency in Python or Scala for data engineering and pipeline development
• Hands-on experience with Azure, AWS, or multi-cloud Databricks deployments
• Knowledge of data storage layers (Azure Data Lake Storage, AWS S3) and performance trade-offs
• Version control (Git, GitHub Actions) and CI/CD practices for Databricks
• Bachelor in Computer Science
• 3–7 years of data engineering experience with at least 2 years in Databricks-focused tuning
• Databricks Certified Data Engineer (Professional) or equivalent preferred

• Collaboration
• Analytical thinking
• Problem-solving
• Databricks
• Apache Spark (SQL, PySpark, Scala)
• Delta Lake

…