Senior Databricks Engineer

Senior Databricks Data Engineer | 12 Month Contract | Data Transformation Programme | Remote (Europe) Start ASAP · Insurance sector The projectAn established insurance business is midway through a large scale data transformation programme, moving off a fragmented estate of legacy warehouses, overnight batch jobs and spreadsheet dependencies, and onto a single governed lakehouse platform built on Databricks. This is not a lift and shift. The programme is rebuilding how the business gets its data: how policy and claims information lands, how it is modelled and quality checked, how actuarial and pricing teams consume it, and how regulatory reporting is produced and evidenced. The target state is a properly governed medallion architecture with lineage, access control and data quality built in from the start rather than bolted on afterwards.The platform team is already in place and delivering. What they need now is additional senior engineering capacity to take ownership of significant pieces of the build over the next twelve months. Why this role is worth a lookYou are joining a transformation that has executive sponsorship, a funded roadmap and a clear target architecture, which is a meaningfully different experience from a programme still arguing about its direction. The work is genuinely greenfield in places: new ingestion patterns, new data products, new governance model. Elsewhere you will be decommissioning legacy pipelines and proving equivalence against the old outputs, which in a regulated insurance context requires rigour rather than speed.It is also a role with breadth. You will not be handed a narrow backlog of tickets. You will be modelling domains with the business, making architectural calls, setting standards for how the team builds, and acting as a point of reference for less experienced engineers around you. What you will be doing▸ Designing, building and optimising production grade ETL and ELT pipelines on Databricks using PySpark, Delta Lake and Delta Live Tables ▸ Modelling and implementing the medallion architecture (bronze, silver, gold) across policy administration, claims, actuarial and third party data sources ▸ Migrating and decommissioning legacy warehouse and batch workloads, including reconciliation of new outputs against existing reporting so the business can sign off with confidence ▸ Tuning Spark workloads and cluster configuration for cost and performance, including partitioning strategy, liquid clustering, Z ordering and caching ▸ Implementing Unity Catalog governance end to end: catalogs, schemas, lineage, access controls and data classification appropriate to a regulated insurer ▸ Building CI/CD for data assets using Databricks Asset Bundles or Terraform, with automated testing and clean environment promotion ▸ Embedding data quality, observability and monitoring so problems are caught before they reach actuarial, pricing and reporting consumers ▸ Working directly with actuarial, underwriting and BI stakeholders to turn business requirements into reliable, documented data products ▸ Contributing to architectural decisions, code review and the engineering standards of the wider platform team Three non negotiables✅ You hold a Databricks Professional level certification. Certified Data Engineer Professional or equivalent, currently held and valid. Evidence will be requested at screening stage. ✅ You are resident within Europe. The client requires all contractors to be based in Europe, for working hours overlap and data handling reasons. ✅ You are available to start ASAP. Onboarding is ready to run and the roadmap is already committed. Applications that do not meet all three cannot be progressed, so please do not apply if any of the above is not the case. Essential experience▸ Six or more years in data engineering, with at least three of those delivering production workloads on Databricks ▸ Expert level PySpark and strong SQL. Comfortable reading a Spark UI and diagnosing skew, spill and shuffle problems without hand holding ▸ Deep practical experience of Delta Lake: schema evolution, time travel, optimisation, streaming ingestion ▸ Hands on Unity Catalog implementation experience, not just consumption of a catalog somebody else configured ▸ Cloud platform experience on Azure, AWS or GCP, including storage, networking fundamentals, identity and secrets management ▸ Git based workflow, infrastructure as code and CI/CD for data pipelines ▸ Experience delivering inside a larger transformation or migration programme, with the stakeholder management that implies ▸ Able to operate as an independent contractor through your own company on a B2B basis Desirable▸ Prior delivery in insurance, reinsurance or another regulated financial services environment ▸ Familiarity with insurance data domains: policy administration, claims, exposure, premium and loss reserving ▸ Azure Databricks specifically, plus Data Factory, Event Hubs or equivalent orchestration and streaming services ▸ dbt, Airflow, Kafka or Databricks Workflows at scale ▸ Awareness of Solvency II, IFRS 17 or GDPR obligations as they land on a data platform ▸ Additional Databricks, Azure or AWS data engineering certifications ▸ Track record of mentoring engineers and raising standards across a team The detailsTwelve month B2B contract, with extension likely subject to roadmap and performance. Fully remote with no travel requirement, working standard core hours with reasonable overlap to the client's working day. Contracting through your own limited company or equivalent entity, invoiced monthly. Competitive day rate dependent on experience, discussed on application. Client name shared with shortlisted candidates. ProcessCV screen and qualification call with TechShack, including certification verification, then a technical interview with the client's platform lead, then a final stage covering architecture and stakeholder scenarios. Interview slots are already open and the client is moving quickly. To applyApply here