Databricks Data Engineer

Databricks Data Engineer | 12 Month+ B2B Contract | Fully Remote | €500 per day I'm partnering with a global technology consultancy on a large data transformation programme for a well-established organisation in the insurance sector. They need an experienced, hands-on Databricks Data Engineer to join the delivery team on a long-term contract. The RoleThe end client is moving away from legacy data warehousing and scattered on-premise systems to a modern, unified Lakehouse platform built on Databricks. You will sit at the centre of this transformation. You'll design and build scalable data pipelines, migrate legacy workloads, and help set the engineering standards the wider platform will be built on.You'll work closely with architects, analysts and business stakeholders across the underwriting, claims, policy and finance data domains.This is a hands-on engineering role. You should be comfortable writing production-grade PySpark and SQL every day. The ProjectMigrating legacy data warehouse and ETL workloads (SQL Server, SSIS, Informatica or similar) onto DatabricksDesigning and building a Medallion (Bronze, Silver, Gold) Lakehouse architecture on Delta LakeRolling out Unity Catalog for central governance, lineage and access controlDelivering curated, trusted data products for actuarial, risk, claims and regulatory reportingIntroducing CI/CD, automated testing and infrastructure as code across the platformEnabling downstream BI, analytics and future AI/ML use cases Key ResponsibilitiesDesign, build and maintain scalable batch and streaming pipelines on Databricks using PySpark, Spark SQL and Delta LakeMigrate legacy ETL processes and data models onto the Lakehouse, keeping data accurate and reconciled throughoutImplement Medallion architecture patterns and build reusable ingestion and transformation frameworksOrchestrate workflows using Databricks Workflows, Delta Live Tables, Azure Data Factory or AirflowWork within Unity Catalog to enforce governance, security and lineageOptimise Spark jobs and cluster configurations for performance and costBuild data quality checks, monitoring and alerting into pipelinesContribute to CI/CD and infrastructure as code using Databricks Asset Bundles, Terraform, Azure DevOps or GitHub ActionsTurn business requirements into robust data solutions alongside architects, analysts and insurance stakeholdersDocument solutions and contribute to engineering best practice across the programme Essential Experience5+ years of commercial experience as a Data Engineer2+ years of hands-on Databricks delivery in productionAdvanced Python and PySparkAdvanced SQL and data modelling (dimensional, Data Vault or similar)Delta Lake and Lakehouse / Medallion architectureCloud platform experience, ideally Azure (AWS also considered)Building and orchestrating ETL/ELT pipelines at scaleGit-based workflows and CI/CD for data engineeringExperience on a data migration or transformation programmeConfident communicator, comfortable working on your own in a remote team Desirable ExperienceDatabricks Certified Data Engineer ProfessionalUnity Catalog implementation or migrationDelta Live Tables and Structured StreamingDatabricks Asset Bundles and TerraformInsurance, financial services or other regulated industry experienceInsurance data domains such as policy, claims, underwriting and actuarialExposure to regulatory reporting (Solvency II, IFRS 17)dbt, Azure Data Factory or AirflowData quality frameworks such as Great Expectations or DLT expectationsPrevious consultancy or client-facing delivery experience CertificationThe ideal candidate holds the Databricks Certified Data Engineer Professional certification, and certified candidates will be prioritised. Associate-certified engineers with strong hands-on delivery experience are also encouraged to apply. Why Apply?12 month+ initial contract with a strong chance of extensionFully remote working€500 per day on a B2B basisWork on a near-greenfield build on a modern Databricks Lakehouse platformReal influence over engineering standards and architecture as the platform takes shapeA high-profile programme with a global consultancy and a major insurance client Please note: this role is not suited to purely BI or reporting profiles, or to architects and delivery leads who no longer code day to day. Candidates must be able to work on a B2B contract basis. Contract Details📍 Location: Fully remote 📄 Engagement: B2B contract, 12 months+ 💶 Rate: €500 per day 🚀 Start: ASAP