Data Engineer

Contract Duration: 11 months

Contract Type: Contract

Role Overview

We are looking for an experienced GCP Data Engineer / Technology Lead to lead the engineering delivery and technical direction of a scalable, multi-tenant Data Integration Platform hosted on Google Cloud Platform (GCP).

The platform uses Apache Airflow to orchestrate ETL pipelines, moving data from source systems through the platform to downstream vendor systems. The successful candidate will provide hands‑on technical leadership across data engineering, cloud architecture, automation, CI/CD, data quality, observability and end‑to‑end data lineage.

This is a hands‑on technical leadership role requiring someone who can combine strong engineering capability with architectural thinking and stakeholder management.

Key Responsibilities

Technical Leadership & Architecture

• Own and drive the technical roadmap in line with business priorities and operational resilience.
• Lead solution design for new integrations, platform capabilities and technical improvements.
• Define and maintain engineering standards covering coding, testing, documentation, observability and security.
• Lead technical design reviews, code reviews and engineering discussions.
• Provide hands‑on technical support for critical platform components, ETL pipelines, design spikes and performance issues.
• Mentor engineers and raise engineering standards across the team.
• Design, develop and maintain production‑grade ETL/ELT pipelines using Apache Airflow.
• Manage pipeline scheduling, dependencies, retries, backfills and failure handling.
• Develop reusable patterns for onboarding new source systems and delivering data to downstream vendors.
• Build reliable and scalable data processing solutions across development, test and production environments.
• Implement data validation, reconciliation and exception‑handling mechanisms.
• Develop operational runbooks and processes to support production data pipelines.
• Design and deliver scalable data solutions using Google Cloud Platform (GCP).
• Work with GCP services including BigQuery, Cloud Storage, Pub/Sub, Dataflow, GKE and/or Cloud Run.
• Apply appropriate GCP IAM, networking, secrets management and environment separation practices.
• Optimise platform performance, reliability and cloud costs.
• Support capacity planning and the delivery of solutions against defined SLAs.

Multi-Tenant Platform Engineering

• Design and maintain scalable multi-tenant platform architecture.
• Implement appropriate tenant isolation and configuration management.
• Develop metadata-driven pipeline and integration frameworks.
• Ensure platform components can scale reliably as new tenants and integrations are onboarded.

CI/CD & Infrastructure

• Design and maintain CI/CD processes for platform services and Airflow DAGs.
• Implement automated testing, static analysis, security scanning and quality gates.
• Support versioning, packaging, deployment promotion and release controls.
• Implement infrastructure as code using Terraform.
• Establish safe deployment, rollout and rollback processes.

Data Lineage, Quality & Governance

• Implement metadata capture and visual data lineage capabilities.
• Ensure appropriate data quality, auditability, retention and access-control practices.
• Work with data governance and architecture teams to improve transparency and traceability.

Reliability & Observability

• Design resilient pipeline patterns using retries, idempotency, dead‑letter mechanisms and backpressure.
• Implement effective logging, metrics, alerting and operational dashboards.
• Identify and resolve performance and reliability issues.
• Drive improvements to incident recovery, operational resilience and MTTR.

Stakeholder Management

• Work closely with product owners, data owners, architects, vendors and governance teams.
• Translate complex technical concepts for both technical and non‑technical stakeholders.
• Lead technical discussions and drive alignment across multiple teams.
• Participate in Agile ceremonies, planning sessions and technical prioritisation.
• Proven experience as a Technical Lead, Lead Data Engineer or Lead Engineer delivering cloud‑based data engineering or integration platforms.
• Strong hands‑on experience with Apache Airflow and production ETL/ELT pipelines.
• Strong experience with Google Cloud Platform (GCP).
• Practical experience with GCP data services such as BigQuery, Cloud Storage, Pub/Sub, Dataflow, GKE and/or Cloud Run.
• Experience working with APIs, configuration management, automated testing and performance tuning.
• Strong experience with CI/CD, DevOps and automated quality gates.
• Practical experience with Terraform / Infrastructure as Code (IaC).
• Strong understanding of cloud security including IAM, networking and secrets management.
• Experience designing reliable data pipelines using patterns such as retries, idempotency, dead‑letter handling and backpressure.
• Ability to communicate technical concepts clearly and work effectively with mixed technical and business audiences.
• Proven experience providing technical leadership, mentoring or guiding engineering teams.
• Experience with data lineage and metadata platforms such as OpenLineage, DataHub or Data Catalog‑style tooling.
• Experience with tenant isolation and metadata-driven frameworks.
• Knowledge of data governance, data quality, auditability, retention and access controls.
• Experience integrating with external vendors and managing technical dependencies.
• Experience working within large‑scale enterprise data environments.

#J-18808-Ljbffr…