Data Engineer

Data Engineer position at BT Group, responsible for designing, developing, and maintaining end-to-end data pipelines using Ab Initio, Apache Spark, and Apache NiFi. Opportunity to work on exciting data products and explore new technologies.

Requirements

• Proficient in building Big Data solutions using Hadoop or GCP or AWS.
• Skilled in technologies such as Scala, Java, Python, Spark (PySpark/Scala), Kafka, Kinesis, BigQuery, Dataflow, BigTable, Terraform, and SQL.
• Experience with Terraform, Kafka, Kinesis, SQL are nice to have.
• Strong expertise in Ab Initio (GDE, PDL, Conduct>IT, Express>IT, EME) with a focus on parallel processing and component optimisation.
• Hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and performance tuning.
• Practical experience with Apache NiFi for flow design, processor tuning, error handling, and security configurations.
• Good understanding of Infrastructure as Code toolsets.
• Experienced in implementing Continuous Integration and Deployment strategies.
• Proficient in using Atlassian tools such as Confluence and JIRA.
• Experienced in version control using Git or SVN.
• Able to clearly present technical issues, progress, and outcomes to team members and Product Owners.
• Demonstrates enthusiasm for learning and applying new technologies.
• Experience or interest in managing and developing team members.
• Capable of building new solutions as well as supporting existing data products and platforms.

Benefits

• 10% on target annual bonus
• Access to an online private GP 24/7 for you and your immediate family
• Market-leading paid carers leave with up to 2 weeks off
• Equalized maternity, paternity, and adoption leave – 18 weeks’ full pay and 8 weeks’ half pay
• Discounted EE and BT products, including mobile and broadband
• Market leading Pension scheme – 5% from you and 10% from us
• Holiday purchase scheme