Data Processing Data Engineer
Titchfield - Working remotely for the foreseeable
You will be working within the Transformation Directorate as part of a team changing the way we collect, process and manage data through surveys and the Census at the core of which is working with new technology in centralising the processing and development of a wide range of internal and external data. You will be contributing to the wider "Statistics for the Public Good" strategy which is at the centre of making maximum use of the large volumes of the data are receiving from both government and other external stakeholders.
The job will entail the development of a number of technical elements covering end-to-end development of large-volume administrative and commercial data, hence experience working in one or more of the following languages would be desirable: Python, SQL (Hive) and Spark as well as experience of manipulation and development of large-scale administrative data.
- Working collaboratively with relevant experts to ensure corporate standards are understood and implemented in all development activity
- Work with subject matter experts to design, build, test and implement processing solutions for social survey data and/or the 2021 Census that will provide high quality data for analysis. This will include working with Data Architecture to ensure data are referenced to core reference dataset.
- Work with Data Architecture to ensure data are fully documented using corporate tools and standards.
- Coordination of across the team, delivery partners within our organisation and end users of the data.
- Manage a small team of technical staff dedicated to the project, this may involve matrix management
- Ensure business continuity plans are developed together with supporting documentation so that assurance can be given around to continuity of data provision
- Intermediate to advanced understanding SQL particularly in the context of analysing data in Hive and Impala;
- An intermediate or better understanding of Python;
- Familiarity with data tools such as Spark;
- Demonstrable experience of structuring and linking within and across very large administrative