Senior Data Engineer

Duration/Location

• Starting date Immediate or as agreed

• Duration 12 months (with possibility of renewal)

• Location France (Paris)

• Status Residential

Context & Environment

The eSpark Project is part of Company's digital transformation in the energy sector, focused on harnessing advanced data solutions to optimize business operations. As part of this initiative, the Senior Data Engineer will join a multidisciplinary team to design and implement data ingestion pipelines using AWS Glue and Lambda, optimizing data processes through PySpark and best practices. The role is pivotal in maintaining high standards of data quality, modular development, and efficient orchestration of data pipelines.

The position requires collaboration across teams to implement scalable and reliable data solutions while adhering to Company's security and quality standards.

Activities

• Data Ingestion and Optimization
  • Design and implement data ingestion pipelines using AWS Step Functions, Glue, and Lambda.
  • Optimize pipelines using PySpark techniques such as caching, persisting, partitioning, and schema inference.
• Data Quality and Governance
  • Implement and maintain robust data quality policies.
  • Develop modular solutions to encapsulate data environments.
• Pipeline Orchestration and Code Quality
  • Manage ingestion pipelines using GitHub Actions, triggers, workflows, and multi-branch setups.
  • Ensure code quality using tools such as Checkmarx and SonarQube.
• Development and Testing
  • Develop new features such as data enrichment functions and monitoring tools.
  • Perform technical validation tests, including unit and integration tests.
  • Deliver technical documentation, including schemas, table structures, and versioning processes.
• Collaboration and Version Control
  • Validate new code versions, ensuring branch management using Git.
  • Experience with branch management in Databricks is a plus.

Qualifications & Experience Required

Must-Have

• Experience: At least 6 years in data engineering.
• AWS Expertise: Proficiency in AWS services including EC2, S3, Lambda, Step Functions, and CloudWatch.
• S3: Data storage and retention policies.
• CloudWatch: Monitoring pipelines and processes.
• Glue: Extensive experience is mandatory.
• Programming Languages: Advanced proficiency in Python and PySpark.
• Version Control: Strong command of Git, including multi-branch workflows, triggers, workflows, and jobs.

Additional Skills (Nice-to-Have)

• Experience with AWS SNS, Inspector, Secrets Manager, and Athena for querying data.
• Familiarity with branch management in Databricks.

Key Skills & Competencies

• Expertise in data ingestion, optimization, and orchestration.
• Strong knowledge of data quality governance and modular development.
• Ability to document technical processes, schemas, and table structures.
• Proficiency in Git and code validation tools (Checkmarx, SonarQube).
• Experience working in a fast-paced, collaborative environment with cross-functional teams.