Data Engineer

Data Engineer | Azure, Databricks & Apache Spark We are looking for an experienced Data Engineer to design, develop and optimise scalable, cloud-based data platforms using Azure, Databricks, Apache Spark and Delta Lake.The role involves end-to-end ownership of data engineering solutions, from architecture and pipeline development to deployment, monitoring and production support. Location: MalmoContract Duration: 1 year + extensionsMode of work: Hybrid Key ResponsibilitiesDesign and develop scalable batch and streaming data pipelines.Build and optimise data solutions using Azure Databricks, Apache Spark and Delta Lake.Develop and maintain data ingestion, transformation and serving layers.Implement lakehouse architectures and data models.Manage Databricks notebooks, job orchestration and performance optimisation.Work with Azure Data Lake Storage (ADLS) and cloud-native data services.Implement CI/CD pipelines, automated testing and Infrastructure as Code.Ensure data quality, reliability, monitoring and cost efficiency.Support production troubleshooting and continuous platform improvements. Key RequirementsStrong hands-on experience with Azure Databricks, Apache Spark and Delta Lake.Proven experience developing end-to-end batch and streaming data pipelines.Strong understanding of lakehouse architecture and data modelling.Experience with Microsoft Azure and Azure Data Lake Storage.Hands-on experience with Databricks notebooks, job orchestration and performance tuning.Strong Git, CI/CD, automated testing and Infrastructure as Code experience.Experience with data validation, monitoring and production troubleshooting.Ability to contribute to solution architecture and technical design.Strong communication and stakeholder collaboration skills.