Senior Data QA Engineer

Job Title: Senior Data QA Engineer
Contract Type: Permanent
Location: London, England
Salary: £23 - £65000.00 per annum
Start Date: ASAP
Reference: BBBH20134_1566909870
Contact Name: Katherine Clarke
Contact Email:
Job Published: August 27, 2019 13:44

Job Description

Role: Senior Data QA Engineer

Salary: £45,000 - £65,000 per annum

Contract type: Permanent

Location: London

I am looking for a Senior Data QA Engineer to join our team in London, leading the modernisation of the organisations data management services, through the development of an Enterprise data platform.

You will be working in a multi-displinary Agile Team. By using Agile approaches to delivery, the team 'work out loud' encouraging open and inclusive discussions to drive development forward.

The Senior Data QA Engineer will be responsible for the testing and validation of data pipelines and showcase to subject matter experts to learn whether it meets their business outcomes. You will be expert at working in cloud environments, such as on Hortonworks and Cloudera based Hadoop clusters. Assurance of data security and privacy in the platform will be relevant to the data pipelines you validate. You will draw on your experiences to engage in discussion with the technical design lead and other technical staff, recommending options for continuous improvements to testing methods.

Key responsibilities

  • Validate software delivered by Data Engineers against user story requirements
  • Ensure principles of privacy and security are assessed and designed into your data solutions
  • Write high quality, generalised test datasets and test methods for verifying and validating new functional and non-functional software, enabling automated testing wherever possible
  • Generate golden datasets to efficiently stress-test software and assure version control
  • Report QA outcomes to Data Engineers and Business Owners
  • Work in an agile way and to do so closely with colleagues
  • Develop new and maintain existing functionality to QA testing for the Data Services platform
  • Provide documentation and training material to support knowledge retention
  • Maintain a high-quality audit trail of your software testing

Key outputs and deliverable

  • Software has had an auditable trail of quality assurance testing conducted on it in accordance with Ofgem data governance expectations
  • Business solutions comply with Privacy and Security requirements
  • Knowledge sharing and best practice training are provided to colleagues and the wider base of stakeholders as required
  • Data product owners are satisfied that code has been robustly demonstrated to meet their needs
  • Documentation regarding QA testing is maintained to a high standard
  • Ensure that governance rules, particularly security standards are always upheld
  • Key stakeholder relationships
  • Collaborate with your colleagues in the data engineering community, discussing and designing software solutions, coordinating during their delivery, mentoring, and sharing best practices
  • Work with the rest of the Data Services community and its key stakeholders, such as Business Analysts and Subject Matter Experts as you review work and demonstrate the quality of data pipelines and quality assurance handling scripts delivered.
  • Support the wider user base with technical support and training, as appropriate
  • The Data privacy and information security teams, and assure requirements are identified and addressed. Creating processes, automated tools and technics to value, protect and assure data services

Specific skills and capabilities:


  • Understanding of Quality Assurance data practices
  • Awareness of OWASP top 10 security considerations and associated tools
  • Hands-on experience writing unit tests and mocking the data
  • Hands on experience to analyse regression reports, automated builds and KPI's
  • Experience of Test-Driven Development applying pair programming and refactoring principles
  • Strong experience of writing BDD acceptance criteria or scenarios
  • Participated in full development data lifecycle
  • Knowledge of software development experiences
  • 3+ years' experience in one of JS/Java/C# programming languages and must have desire to develop strong experience in SQL/HQL for complicated queries
  • Automated test framework development from scratch using Python
  • Experience or at least keen interest in using big data technologies like PySpark/Yarn and Hive
  • Hands on experience with Linux and Microsoft Azure cloud environment or equivalent
  • Experience in using considerable data sizes (TB/PB)
  • Exposure to continuous build and deployment process, with hands on experience in using build systems such as Oozie/Jenkins
  • Familiar with defect management systems either using JIRA or similar
  • Willingness to constructively challenge the work of others, to ensure high standards are achieved
  • Experience of carrying out showcase acceptance demonstrations with your stakeholders
  • Proven experience of innovation and improving best practice


  • Experience in Agile approaches to working
  • Experience with performance tuning in Hadoop configuration systems
  • Knowledge of Business Intelligence tools, such as QlikSense
  • Experience in accessibility, stress, performance or load testing
  • Working knowledge of HBase, Solr, Kafka and Flume

Benefits include:

  • 25 days holiday, rising to 30 after 2 years of service
  • Competitive Pension
  • Childcare Vouchers
  • Subsidised gym membership
  • Season Ticket Loan Scheme
  • Cycle to Work Scheme
  • Flexible working hours and flexi-time

If this role sounds like you then please apply ASAP or give me Katherine Clarke a call on 01173320813 or email me on Katherine. Clarke @