Data Engineer – Commodities

Overview

As a Data Engineer in the Commodities Technology team, you design and operate end-to-end ETL pipelines to ingest and transform commodities data from multiple sources. You build standardized data models and metadata to enable discovery and reuse, and you ensure reliable delivery with workflow orchestration. You apply AI to automate schema inference, monitor data health, and maintain clear data lineage and documentation. You work closely with Portfolio Managers, researchers, and data strategists to evolve datasets and support trading-focused analytics.

Responsibilities

• Design, implement, and maintain end-to-end ETL pipelines in Python and SQL for multiple data sources
• Build and maintain standardized data models, schemas, and metadata
• Schedule and monitor data pipelines using Airflow or similar tools
• Implement validation, reconciliation, and anomaly-detection controls
• Apply AI to automate schema inference and manage schema drift
• Use AI-driven data-quality, observability, and documentation capabilities for data health and lineage
• Maintain production-ready code and repeatable deployments with Git, GitHub Actions, and PyTest
• Collaborate with Portfolio Managers, researchers, data analysts, and data strategists to refine datasets and documentation

Key requirements

• 5+ years of data engineering or analytics engineering experience
• Proficiency in Python and SQL with large datasets and complex transformations
• Hands-on experience with Airflow or equivalent workflow tools
• Experience with Git, CI/CD (GitHub Actions), and PyTest or similar testing frameworks
• Strong attention to detail, data quality, and documentation
• Ability to work independently and communicate with diverse stakeholders
• Knowledge of commodities markets and data (preferred)
• Experience with data warehousing, columnar storage, analytics databases, data catalog or governance tools (preferred)

• attention to detail
• effective communication
• independence
• Python
• SQL
• Airflow

…