- Newport - Currently Remote working
- £23 - £26 per hour rate (Inside IR35) - Self-employed contract
- Experience with Cloudera, ETL, SQL, Python
We are recruiting for a suitably skilled data engineer to provide central support to the Core Data Engineering Team around the development and processing of data deliveries from another government department as well as proving programming and data engineering support around the delivery of a number of elements of business survey redevelopment to make surveys more responsive to the economic disruption arising from contemporary events.
This will involve developing operational and data to form the basis of a series of products as well as applying established data engineering and data modelling methods to the data for the main stage of the product build.
You will be using key programming languages such as Python (through Spark), Scala (through Spark) and SQL (through Hive and Impala) in a big data context.
- Development of ETL methods for a range of internal and external data sources converting thee from unstructured formats to dynamic tables and views in Hive and Impala;
- Providing coding support and coaching to a growing team of data engineers sharing best practice and established methods.
- Assisting in the development of analytical layers of data from raw HDFS files for use in producing a range of outputs.
- Extensive proven experience of data engineering and architectural techniques, including data wrangling, data profiling, data preparation, metadata development, and data upload/download;
- Proven experience of 'big data' environments, including the Hadoop Stack (Cloudera), including data ingestion, processing and storage using HDFS, Spark, Hive and Impala;
- Extensive hands-on experience of developing ETL functionality in a cloud or on-premise environment;
- Experience of using tools such as python and SQL (in Spark) to profile, query and structure large-volume data;
- Proven experience of using Cloud Services particularly in the context of Hadoop;
- Experience of developing/utilising programming and query languages e.g. SQL (Hive Impala specifically), Python (through Spark), Scala.
- Understanding of data bases and applying data models in relational database formats.
- Experience of coaching and training others in programming and ETL techniques;
- Experience of UK Government, particularly HMRC Administrative Data;
If this is a role you would be interested in applying for, please apply with an up-to-date CV asap.