Azure Data Engineer

Location: 1 x per week in Central London
Contract Type: Outside IR35
Duration: 6-months
Rate: £450 per day

About the Role:

We are looking for a highly skilled Azure Data Engineer with expertise in Databricks and Lakehouse architectures to join a cutting-edge data transformation project. You will play a key role in designing, building, and optimizing data pipelines in an Azure cloud environment, ensuring scalability, performance, and cost-efficiency.

Key Responsibilities:

• Develop and maintain data pipelines using Azure Data Factory (ADF), Databricks, and Delta Lake.
• Implement Lakehouse architecture principles, ensuring a scalable, governed, and cost-efficient data ecosystem.
• Optimize ETL/ELT processes using PySpark, SQL, and Databricks workflows.
• Design and implement real-time and batch data ingestion from various sources (structured/unstructured).
• Work with Azure Data Lake Storage (ADLS), Azure Synapse, and Azure SQL for data processing and analytics.
• Enhance data governance, security, and compliance using RBAC, encryption, and masking techniques.
• Support performance tuning and optimization of queries, storage, and processing pipelines.
• Implement CI/CD pipelines for automated deployment and version control of data workflows.
• Collaborate with data architects, BI analysts, and business stakeholders to align solutions with business objectives.

✔ Strong hands-on experience in Azure Data Engineering with expertise in Databricks, Delta Lake, and ADF.
✔ Proficiency in PySpark, SQL, Scala, and Python for data transformations and analytics.
✔ Deep understanding of Lakehouse architecture, Medallion patterns, and Delta Lake optimizations.
✔ Experience with Azure Data Lake (ADLS), Azure Synapse, and Azure SQL.
✔ Knowledge of data governance, security best practices, and compliance frameworks (GDPR, ISO, etc.).
✔ Experience with event-driven architectures, including Azure Event Hub, Kafka, or streaming technologies.
✔ Hands-on expertise in CI/CD pipelines for data solutions using Terraform, ARM templates, or Azure DevOps.
✔ Strong problem-solving skills, with a focus on performance tuning, partitioning, and indexing strategies.
✔ Ability to work in agile, fast-paced environments and collaborate with cross-functional teams.

Nice to Have:

• Familiarity with Machine Learning and AI-driven analytics in Azure ML.
• Experience with Power BI, DAX, and advanced data visualization.
• Knowledge of multi-cloud data architectures (AWS, GCP).