Overview
In this Senior Data Engineer role at a fast-growing UK-based software business, you will build scalable data solutions in Databricks to establish a single source of truth and empower trusted self-service analytics. You will own end-to-end data pipelines and collaborate across Finance, Product, Sales, Marketing and Engineering to translate business needs into technical solutions. You will shape the enterprise data architecture with a focus on scalability, security and performance, and embed data quality and governance across the data lifecycle. You will mentor a junior engineer and contribute to maturing a growing data function, gaining visibility into data strategy and platform evolution.
Pay / Benefits
- health insurance
- income protection
- life assurance
- subsidised gym membership
- 25 days’ holiday
- birthday off
Responsibilities
- Design, build and maintain scalable data pipelines and data products using Databricks, PySpark, Spark SQL and Delta Lake
- Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems
- Shape enterprise data architecture, balancing scalability, performance, security and maintainability
- Partner with stakeholders across Finance, Product, Sales, Marketing and Engineering to translate business challenges into technical solutions
- Embed data quality, governance and access controls throughout the data lifecycle
- Mentor a junior engineer and help mature a growing data function
Key requirements
- Senior level experience in data engineering with strong Databricks skills (PySpark, Spark SQL, Delta Lake)
- Advanced SQL and Python coding skills
- Track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions
- Solid grounding in data architecture, modelling and modern data platform design
- Cloud experience, preferably AWS (Azure also acceptable), and familiarity with CI/CD and source control best practice
- Stakeholder collaboration
- Mentoring and coaching
- Problem solving and analytical thinking
- Databricks
- PySpark
- Spark SQL
…