Senior Data Engineer (with Databricks)

Project Overview: Our client is embarking on a significant transformation project aimed at replatforming their financial system.

This ambitious initiative includes two parallel efforts: one focusing on all financial data and the other on non-financial data.

We are in the early stages of consolidating data from various sources into a single Delta Lake data warehouse, and we are looking for a Senior Data Engineer to lead this critical endeavor.

Responsibilities:

• Lead the integration and ingestion of data from diverse sources into the Delta Lake platform, ensuring data consistency and availability.


• Collaborate with various teams to identify all data sources, including SFTP/flat files, and APIs.


• Design and implement systematic processes for data ingestion and integration.


• Conduct extensive data analysis to support the consolidation of organizational data into a single data warehouse.


• Utilize Databricks for data ingestion and reporting, transitioning data from multiple data warehouses to a unified Delta Lake platform.


• Work closely with the development team to understand project scope, gather full requirements, and plan subsequent phases.


• Participate in daily standups, sprint planning, backlog refinement, and retrospectives within a scaled scrum framework.


• Exhibit flexibility and adaptability in navigating unknowns and evolving project requirements.

Requirements:

• Bachelor's or Master's degree in Computer Science, Data Science, or a related field.


• Solid understanding of the software development life cycle.


• Proven experience with Python, Spark, and SQL for data engineering and analysis.


• Strong data analysis skills, with the ability to interpret and manipulate large datasets.


• Experience with Databricks for data ingestion and transformation.


• Familiarity with Delta Lake and data warehousing concepts.


• Excellent problem-solving skills and the ability to see the big picture and understand the broader context of assigned tasks.


• Strong teamwork and collaboration skills, with a demonstrated ability to work effectively in a team-oriented environment.


• Knowledge of reporting tools like Power BI.


• Knowledge of Data Science concepts is a plus.


• Strong communication, interpersonal, and presentation skills.

Start Date: ASAP
Location: Remote
US Hours Needed: Minimum until 6pm CET
Type of Employment: B2B