Lead Data Architect – Insurance Data & Analytics (Databricks Migration)
London (Hybrid) |Permanent, Full-Time
About the Role
Our client, a leading provider of benchmark data and analytics for the financial services sector, is hiring the first senior engineering leader for their UK Insurance data business. This is a newly created role, not a backfill.
This is a rare opportunity to own a platform transformation from the ground up: migrating a legacy SQL benchmarking engine onto a modern Databricks Lakehouse, and then building the product tooling on top of it.
Our client provides benchmark data behind pricing, acquisition, renewal, and retention decisions across the UK insurance market, built on a proprietary, behavioural dataset that underpins trusted relationships with senior commercial decision-makers across the industry.
You’ll report directly to the Head of Insurance, based in London, and will be the senior technical voice for the UK Insurance business – with the option to line-manage an existing mid-level engineer as the team grows. You’ll also work closely with a broader, global engineering function on platform standards, while owning delivery locally.
What You’ll Do
- Lead the migration of legacy SQL solutions into Databricks Jobs, Workflows, and PySpark/SQL pipelines, starting with a lift-and-shift of core benchmark data, while maintaining operational SLAs throughout
- Define the target-state data architecture (Lakehouse, Medallion, domain-oriented data products) to support reporting, analytics, and future AI/ML use cases
- Work directly with business leadership and analysts to codify canonical metric logic – replacing spreadsheet calculations with documented, testable code
- Own UK Unity Catalog governance: catalogue, schema, table design, ownership, and access control
- Build automated QA and reconciliation between legacy and new outputs, with the observability to give internal and external stakeholders confidence in the data
- Implement CI/CD and Infrastructure as Code for Databricks assets
- Tune Spark and SQL workloads for performance and cost
- Act as the primary link between UK delivery and the centralised platform engineering function
- Provide technical leadership and mentorship, with the option to line-manage a mid-level engineer
What We’re Looking For
- Hands-on experience delivering a Databricks migration at production scale – this is the core of the role
- Strong stakeholder and relationship management skills. You’ll work closely with business leadership, analysts, and engineers across regions, often with ambiguity to navigate
- Advanced Python, PySpark, and SQL
- A track record migrating SQL Server workloads, stored procedures, and scheduled jobs to modern, cloud-native platforms
- Solid understanding of Unity Catalog governance, security models, and access control
- Experience with CI/CD, automated testing, and Infrastructure as Code
We’d love if you also have:
- Experience in a smaller, B2B-focused fintech environment – this role suits someone used to close commercial context and hands-on delivery, rather than large enterprise-scale environments with extensive legacy system sprawl
- Insurance, financial services, pricing, or marketing analytics background
- Familiarity with UK GDPR, data residency, and regional governance
- dbt or analytics engineering experience
Why This Role
This isn’t a startup. It’s an established, well-regarded business in its space, but you’ll have the ownership and autonomy of building something from scratch. You’ll shape the architecture, the standards, and potentially the team, with the backing and technical standards of a wider global engineering organisation behind you.
#J-18808-Ljbffr…