Principal Data Engineer

About Digital Futures

Digital Futures is an AI services and software company that specialises in accelerating AI transformation and enabling AI governance for enterprise and government.

The Role

Digital Futures is supporting a large, regulated organisation to build and operate trusted, scalable data assets that enable priority business use cases across its enterprise data platform. This is a hands-on engineering role, moving use cases from requirements into production-ready, reusable and governed data assets.

Working as part of a multidisciplinary team of data analysts, engineers and data scientists, you will design, build, test, deploy and support scalable data solutions, turning agreed requirements into robust data pipelines, curated datasets and data products. You will work closely with business stakeholders, product and delivery leads, data owners, architects and platform teams and, as a principal-level engineer, you will shape technical designs and set engineering standards for the team.

The role combines strong big-data and cloud engineering capability on GCP or AWS with a practical understanding of data modelling, data quality, Agile delivery and production-ready DevOps practices.

Key Responsibilities

Data engineering and solution delivery

• - Design and build reliable, scalable and maintainable data pipelines, data products and processing solutions, leading solution designs that are secure, observable, reusable and aligned to enterprise standards.
• - Translate requirements, source-to-target mappings and acceptance criteria into technical designs and production-ready implementations across batch and, where relevant, event-driven patterns.
• - Support the full lifecycle of data assets, including development, testing, deployment, monitoring, incident resolution and continuous improvement.

Big-data and cloud platform engineering

• - Build and maintain distributed data-processing solutions using Hadoop, Hive, HDFS, Apache Spark and Scala or Python.
• - Develop and deploy data solutions on GCP (such as BigQuery, Dataflow, Dataproc, Pub/Sub and Cloud Composer) or AWS (such as Redshift, Glue, EMR, Kinesis and Managed Workflows for Apache Airflow), including migrating and modernising workloads onto the cloud.
• - Select processing, storage and orchestration patterns based on data volume, latency, reliability, cost, security and operational requirements.

Data modelling, quality and controls

• - Apply relational and non-relational data-modelling techniques, creating schemas, tables, views and partitions that support performance, usability and maintainability.
• - Implement data-quality checks covering completeness, validity, uniqueness, consistency, timeliness, reconciliation and referential integrity.
• - Ensure lineage, ownership, access, privacy, retention and permitted-use considerations are reflected and documented in technical delivery.

DevOps, testing and production readiness

• - Implement automated unit, integration and data-quality testing for data-processing logic and platform components.
• - Contribute to automated build, deployment and release processes using Git or GitHub and tools such as Jenkins and Ansible, applying peer review, environment promotion and rollback practices.
• - Monitor pipeline health, data freshness and performance in production, maintaining runbooks and support procedures, and investigating code and data issues to identify and communicate root causes.

Agile collaboration and delivery

• - Work within Scrum, Kanban or a comparable Agile model, using Jira to manage stories, dependencies, defects and delivery status.
• - Break complex engineering work into deliverable increments, providing realistic estimates and technical sequencing and managing technical risks.
• - Collaborate with data analysts, data scientists, delivery leads and business stakeholders, mentoring engineers and contributing to a collaborative engineering culture across global teams.

Experience and Profile

• - Extensive experience in data engineering, big-data engineering, cloud data engineering or a related role, typically 5+ years.
• - Strong hands-on experience with Hadoop, Hive, HDFS, Apache Spark, Scala, SQL and Airflow.
• - Hands-on experience with data migration and processing on GCP or AWS, using services such as BigQuery, Dataflow, Dataproc and Pub/Sub, or Redshift, Glue, EMR and Kinesis.
• - Proven ability to design and deploy large-scale distributed data-processing systems, defining architecturally sound solution designs and understanding the logical flow of solutions at code level.
• - Experience delivering automated unit testing, integration testing and automated build and deployment processes, using Git or GitHub, Jenkins, Ansible and Jira.
• - Understanding of big-data modelling using relational and non-relational approaches, including PostgreSQL or equivalent databases and enterprise data warehouses.
• - Experience debugging code and data issues, analysing differences and communicating root causes and remediation options.
• - Understanding of cloud design patterns, cloud security principles, observability and production support.
• - Strong interpersonal skills, with the ability to work effectively with senior business stakeholders in team-based and global environments, and a proactive, learning mindset.
• - Experience in financial services or another regulated environment, and a GCP or AWS certification, is desirable.

Benefits

We offer a competitive salary. Our benefits package is designed to support your wellbeing and development. This includes:

• - Group Life Assurance
• - Virtual GP and Online Physiotherapy
• - Confidential Helpline/Employee Assistance Programme
• - Wellbeing Studio & Digital Gym
• - Discounted shopping cards, e-vouchers & cashback offers
• - Percentage discounts & savings on brands online & in store
• - Statutory pension contribution - salary exchange scheme
• - 30 days annual leave, including UK Bank Holidays

Why Join Us

Digital Futures consultants deliver solutions across the full data and AI transformation lifecycle, from strategy, analytics and engineering through to adoption, governance and assurance.

We’re focused on creating opportunity in the age of AI. Joining Digital Futures means joining an organisation making a real impact in delivering an AI-first future, at whatever stage of your career.