Senior Data Engineer - Big Data

Job Title: Senior Data Engineer - Big Data
Contract Type: Permanent
Location: London, England
Salary: £50000 - £75000 per annum + Pension, Flexi time, gym
Start Date: ASAP
Reference: BBBH20551_1571413578
Contact Name: Eleanor George
Contact Email:
Job Published: October 18, 2019 16:46

Job Description

Senior Data Engineer

We are looking for a Senior Data Engineer to join the Data Team for this public sector organisation. This is an innovative and forward-thinking team, leading the modernisation of data management for this organisation through the development of an Enterprise data platform. The data provided will be used to make a difference for all citizens across the UK.

The team is comprised of product leads, data scientists, software engineers, user researchers, QA testers, business and data analysts. By using Agile approaches to delivery, the team 'work out loud' encouraging open and inclusive discussions to drive development forward.

The Senior Data Engineer will be responsible for the construction of data products for the business and delivering new functionality to the Data Services platform. You will engage in discussion with technical design leads and other technical staff, recommending options for implementing technical infrastructure and future data products. You will facilitate the end-to-end data handling approach of data pipelines using the Data Services platform. Defining and communicating the technical needs for data collection, ingestion, transformation and calculation.

Specific Skills and capabilities required:


  • Advanced knowledge of software development:
  • The software development lifecycle and specific applications to data engineering, including Test Driven Development and quality assurance methods
  • An understand of software design and architecture best practice
  • Ability to create a code base with a high-test coverage and ensure test coverage remains high in development through automated test processes (for example continuous deployment / continuous integration best practice) Knowledge of data management, handling and processing
  • Data Quality
  • Data Preparation
  • Data Cleansing
  • Data Modelling
  • Experience working with and integrating into Business Intelligence tool
  • Hands on experience in platforms such as Azure and the Hadoop ecosystem (HIVE, HDFS, Apache Spark, Airflow)
  • Advanced competency with Python and SQL. Knowledge of Linux
  • Senior Team member
  • Ability to provide effective knowledge transfer / retention and support others accordingly
  • Proven experience of innovation and improving best practice
  • Experience of collaboration and partnering within a team, and implementing effective data engineering working practices in agile teams


  • Experience working with privacy and information accreditation or assurance teams
  • Experience in the delivery of data solutions capable of:
  • being certified to an information security standard such as ISO27001 or equivalent
  • holding UK Government OFFICIAL-sensitive or equivalent information
  • meeting legal and regulatory requirements such as DPA or GDPR
  • SQL Databases, such as MS SQL and/or Postgress
  • Cloud/Big Data: HDInsight, S3, EMR
  • NoSQL Databases, such as Neo4j

How to apply:

To apply for this role, you are required to submit:

  1. Up to date CV
  2. Suitability Statement - to address the essential and desirable criteria included in the role profile.

Completed applications should be submitted by midnight on the 2nd November 2020

If you have the relevant experience and are up for a new challenge - apply today. I look forward to receiving your application and talking you through this role in more detail.