Senior Python Developer (PySpark)

Job Title: Senior Python Developer (PySpark)
Contract Type: Contract
Location: England
Salary: £650 - £750 per day + Inside IR35
Start Date: ASAP
Reference: BBBH29971_1648826078
Contact Name: James Norval
Contact Email:
Job Published: April 01, 2022 16:14

Job Description

Role Title

Senior Software Engineer/Python Developer

Security Clearance required

SC clearance

Role location

Remote in the UK, travel to site also fine

Duration of engagement

4 months initially


12 Noon 06/04/22

Description of Requirement:
The role will report to the Technical Lead for the team, and work alongside another Senior Python Developer, together with several more junior development staff.
The role is responsible for the technical development and implementation of the data processing pipeline specified and designed in conjunction with the COVID analysis teams. You will be working with a small coding team and a wider analysis team to identify the best ways of ingesting and processing the data received.

Throughout you will drive the delivery of ETL products, whilst supporting team members and others to apply agile and lean principles to deliverables. You will:

  • Contribute to design work both at the product and feature level. You will be expected to lead on design and specification at the feature level, providing guidance to the more junior team members.
  • You will work closely with and provide active input and support for the Technical Lead for the overarching product level design and solution specification.
  • Collaborate with the Team Leader, Product Owner and Scrum Master to support the team in adopting best working practice suitable for a small development team, working to very tight deadlines.
  • Work with others, through pair programming, to implement a data engineering pipelines from the ETL of the data at ingest through to the creation of the final analytical dataset.
  • Provide active leadership and drive for those aspects of work delegated to you.
  • Ensure that pipelines can be adapted, and re-run should data quality issues be identified.
  • Liaise with analysis teams to ensure that requirements for outputs are understood and incorporated into the pipeline.
  • Address technical blockers, actively seeking solutions to remove them and proposing alternative routes to delivery.
  • Routinely test the developed pipelines to ensure that they are fit for purpose.
  • Peer review work of others in the team, to ensure quality and consistency.

As part of the interview process you will be asked to complete a test as below:
"With a focus on communication with other members of the team, you will be asked to discuss and solve a practical problem using PySpark. This will include writing a function and associated unit test."

Relevant Skills and Experience:

Desirable or
  • Extensive Software Engineering practice in a small team environment, utilising Scrum based agile approaches - including applying DevOps practices, including continuous integration and continuous deployment

  • Design and specification skills at both the product and feature level. Data engineering - development of ETL routines for slowly changing dimensions

  • Coding - functional programming in Python & PySpark, multicore/distributed processing (Hadoop), non-relational (Hive/Impala) SQL database systems

  • Testing:

    • unit and integration testing for PySpark routines using pytest
    • Support and bug investigation / fixing for proving / acceptance testing

  • Analysis skills both interacting with stakeholders to establish requirements and problem solving, identifying the root cause of data quality issues and communicating these with data providers / users

  • Strong interpersonal & communication skills, to deal effectively with complex requirements / issues with stakeholders