Job Details

Principal Data Engineer

Join the ECI Group's Professional Technical Services Network as a Principal Data Engineer. This is a Houston based six-month contract role with an anticipated start date of March 30, 2026 and ending on September 30, 2026. You must be a US Citizen or Permanent US resident to qualify for the position.

Roles and Responsibilities

The Principal Data Engineer is the one who conceptualizes and designs data foundations and end to end solutions including data platforms for the Shell Business to maximize value from data. The role helps create a data-driven thinking within the organization, not just within IT teams, but also in the wider business stakeholder community. A Principal Data Engineer is expected to be a subject matter expert, and an active thought partner in the organization's data strategy and the key driver to define the Vision and Data Strategy for robust and future ready IT solutions.

Role Objective

This is not a standard data engineer role. We are looking for a deeply technical, hands-on individual contributor who can:

• Diagnose performance, latency, and cost issues in a large-scale cloud data platform
• Take a top-down, platform-level view across multiple projects
• Improve architecture, efficiency, and cost optimization, not just write Spark code
• Act as a technical problem-solver and mentor, guiding other data engineers

Current Platform & Architecture Data Flow

This person is expected to make the platform better, not just execute tasks.

• On-premise systems ? Cloud (Azure)
• Streaming ingestion ? Azure Data Lake Storage (ADLS)
• Data processed into two separate containers:
• Crude trading
• Product trading

Technologies in Use

• Qlik Replicate (formerly Attunity)
• Azure Data Lake Storage (ADLS)
• Databricks
• Delta Live Tables (DLT)
• Spark / PySpark
• Python
• SQL (complex queries and procedures)

Key Challenges the Role Is Meant to Solve

1. Data Latency

• High-volume streaming data
• End-to-end latency issues that need root-cause analysis

2. Databricks / DLT Cost Spikes

• DLT costs are far higher than expected
• Known contributors:
• Very high data volume (expected)
• Inefficient lookup logic used to split data into separate containers
• The current solution works but is not optimal

Mandatory Skills

• AWS/Azure/SAP Master
• ELT Master
• Data Modeling Master
• Data Integration & Ingestion Master
• Data Manipulation and Processing Skill
• GITHUB, Action, Azure DevOps Skill
• Data factory, Databricks, SQL DB, Synapse, Stream Analytics, Glue, Airflow, Kinesis, Redshift, SonarQube, PyTest Skill

Programming & Data Skills

• Strong Python (mandatory)
• PySpark (advanced, not basic)
• Advanced SQL
• Complex queries
• Stored procedures
• Analytical logic

Optional Skills

• Experience in project management, running a scrum team.
• Experience working with BPC, Planning.
• Exposure to working with external technical ecosystem.
• Managing small teams.
• MKDocs documentation

Portfolio: Shell IDT/Downstream & Renewables IDT/Trading & Supply/TS - Crude & Products/TS - Crude Trading/TS - Crude Investments Program: Project: Crude Shift WONA - Crude Data Exchange Description: Considering the decision to pause the work for the DEX to Endur migration, focus remains on advancing the Crude PDS Transformation, with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of "cleaning and aligning" the data between WONA and NA. Ambition to eventually move NA ODW into this same platform. Key focus Business case: Considering the decision to pause the work for the DEX to Endur migration, focus remains on advancing the Crude PDS Transformation, with a key priority to establish a global view of data with standardized granularity across the T&S Crude Business. This funding request is to start work on modernizing the WONA Data Warehouse using Azure cloud technology with the target of "cleaning and aligning" the data between WONA and NA. Ambition to eventually move NA ODW into this same platform. Business Application : CRUDE AND PRODUCTS DATA EXCHANGE, Crude Data Exchange Project Type : IT Project Project Class : Application