12-month rolling (multi-year programme)
1 day/week – St Paul’s, London
We’re hiring a Senior & Lead Data Engineer to build a Databricks lakehouse platform in a high-performance, business-critical Front office Risk trading environment.
This is a hands-on engineering role focused on building and optimising large-scale distributed data systems.
This is a highly technical team operating at scale, we’re looking for engineers with deep data engineering expertise, strong low-level Spark knowledge, and experience building high-performance systems using modern Databricks and AI-driven platforms.
This is working for a Front office Risk Data team, so you MUST have experience working closely with Front/middle office users and have good domain knowledge covering Market and credit Risk and derivatives
What you’ll do
- Defining technical approach for migrating historic data from existing SQL server instances to Databricks then keeping that data flow on a daily/intraday basis.
- Building pipelines from bronze to silver/gold/platinum
- Curation of data products in the market risk, credit risk and historic market data domains.
- Build and optimise Spark pipelines on Databricks
- Develop a lakehouse platform (Medallion architecture)
- Own data modelling, architecture, and pipeline design
- Work with large-scale data (TB-PB)
- Drive performance, scalability, and reliability in production
What we’re looking for
- Strong experience running Spark workloads in production
- Proven ability to optimise Spark at scale (Tb/PB datasets)
- Solid Python (Scala beneficial, not essential)
- Experience with data modelling and lakehouse architecture
- Ability to debug and improve performance in distributed systems
Important
- Must have recent, hands-on Spark experience
- Databricks strongly preferred (not essential if Spark depth is very strong)
- Experience supporting AI/ML or advanced analytics platforms is a big plus
- Experience in performance-critical environments
Not a fit if
- Primarily BI / reporting focused
- Spark used only at small scale or outside production
- No experience with performance optimisation in distributed systems
Bottom line
We’re looking for engineers who can design, build, and optimise Spark-based systems at scale and operate effectively in a performance-critical environment from day one.
#J-18808-Ljbffr…