Lead Data Platform Engineer – AWS, Spark

Overview

As Lead Data Platform Engineer in Experian’s Credit Services & Verifications, you will provide hands-on technical leadership for a new fraud data capability within the Data Analytics Platform. You will guide a technical team to deliver secure, scalable data pipelines and cloud components, collaborating with central platform groups, data owners, security and compliance. Your role combines data engineering depth with cloud platform experience, DevSecOps practices and team leadership to accelerate fraud data initiatives. You will mentor engineers and promote modern engineering practices to improve delivery speed and quality.

Pay / Benefits

• Great compensation package and discretionary bonus plan
• Pension
• Bupa healthcare
• Sharesave scheme
• 25 days annual leave
• 3 volunteering days

Responsibilities

• Lead and support an agile technical team delivering data platform solutions for fraud-related data assets
• Guide the delivery of AWS-based data pipelines, data lake components and producer-side data platform capability
• Oversee handover, ownership, maintenance and enhancement of pipelines and cloud components developed by the central DAP team
• Lead building, testing and maintaining high-quality data pipelines from source systems such as Snowflake and internal platforms
• Ensure metadata is prepared and published accurately for surface via the DAP catalogue
• Champion DevSecOps, CI/CD, Infrastructure as Code and cloud-native tooling to improve reliability, security and delivery efficiency
• Mentor and coach senior and junior engineers, fostering innovation and collaboration
• Promote automation and AI-assisted development to improve speed, quality and repeatability

Key requirements

• Strong experience building data pipelines, data lake solutions or data platform capabilities
• Strong experience with AWS data engineering services (Glue, S3, Step Functions, IAM, Lake Formation, EC2, Lambda, EventBridge, SQS)
• Strong experience with DevOps and Infrastructure as Code tooling (Terraform, CI/CD, Git, Jenkins)
• Strong experience developing on Apache Spark or Hadoop
• Significant programming experience in Scala or Python
• Experience with Configuration as Code and API integration
• Strong understanding of cloud security, data privacy, access control and governance in data platforms

• mentoring and coaching
• collaboration across cross-functional teams
• ability to lead in an agile environment
• AWS data services (Glue, S3, Step Functions, IAM, Lake Formation, EC2, Lambda, EventBridge, SQS)
• Terraform or CloudFormation
• OpenShift or Kubernetes

…