London | Hybrid, 3 days per week in the office
6-month initial contract
We are looking for an experienced Senior Data Engineer with strong hands-on expertise across Python, Databricks and Spark to join a large-scale data engineering environment in London.
This is a genuinely hands-on role, with around 30–50% of your time expected to be spent coding, solving technical problems and contributing directly to the development and optimisation of data platforms and pipelines.
Alongside the hands-on engineering work, you will take technical ownership across delivery, provide direction to other engineers and work closely with stakeholders to ensure data engineering initiatives are delivered effectively.
Key responsibilities
- Contribute directly to the design, development and optimisation of data pipelines using Python, PySpark and Databricks
- Build and enhance ETL/ELT processes and reusable data engineering frameworks
- Troubleshoot complex data and performance issues and provide hands‑on support with production issues
- Contribute to code reviews, debugging, testing and performance optimisation
- Own delivery of data engineering initiatives, ensuring work is delivered to a high standard and aligned with business requirements
- Support sprint planning, backlog prioritisation and delivery tracking
- Identify and resolve technical risks, dependencies and blockers
- Provide technical direction and architectural oversight across data engineering initiatives
- Design and support modern Lakehouse architectures using Databricks and Delta Lake
- Lead and mentor other data engineers, providing technical coaching and hands‑on guidance
- Work closely with product owners, architects and business stakeholders to translate requirements into effective technical solutions
- Promote best practice across engineering, CI/CD, DevOps, Agile delivery, data quality and governance
What we’re looking for
- 10+ years’ experience across data engineering and/or software engineering
- Proven experience as a Senior Data Engineer, Technical Lead or similar
- Strong, recent and hands‑on Python development experience is essential
- Extensive experience with Databricks, including notebooks, workflows and Delta Lake
- Strong Apache Spark / PySpark experience
- Proven experience building and optimising large-scale ETL/ELT data pipelines
- Strong understanding of data architecture and modern Lakehouse concepts
- Strong SQL skills and experience working with distributed data systems
- Experience with at least one major cloud platform, such as AWS, Azure or GCP
- Experience working in Agile delivery environments
- Ability to balance hands‑on engineering with technical leadership and delivery responsibility
- Experience with Kafka or Spark Structured Streaming
- CI/CD and DevOps practices
- Delta Lake optimisation
- Docker and/or Kubernetes
- Exposure to machine learning pipelines
The key requirement is that you remain hands-on. This is not a pure delivery or management position. You will be expected to write code, solve complex technical problems and contribute directly to the engineering work, while also providing technical leadership and supporting delivery.
Contract: 6 months initially
Location: London, hybrid with 3 days per week in the office
#J-18808-Ljbffr…