Data Engineer-Snowflake Data Lakehouse on AWS,ETL / ELT ,SQL,data pipelines
Must-have:AWSCloudDevOpsDataAISecurity
Requirements:
- Degree in Computer Science, Information Systems, Data Engineering,
- Software Engineering or a related discipline, or equivalent practical experience.
- Approximately 3-6 years of relevant experience in data engineering, data platforms or cloud data environments.
- Strong hands-on proficiency in SQL and SQL scripting, including complex joins, CTEs, window functions, stored procedures and performance optimisation.
- Strong understanding of Data Lake, Data Warehouse and Data Lakehouse architectures, including data ingestion, transformation, modelling and serving layers.
- Hands-on knowledge or implementation experience with Snowflake.
- Experience designing and maintaining ETL/ELT pipelines.
- Working knowledge of AWS, particularly data-related services such as Amazon S3, AWS Glue and IAM.
- Good understanding of cloud architecture, security, access control and data- governance principles.
- Experience working with structured and semi-structured data such as relational data, CSV, JSON and Parquet.
- Ability to troubleshoot data pipelines and investigate data-quality and performance issues.
- Ability to document architectures, pipeline logic, data flows and operational procedures.
Key Responsibilities
- Data Lakehouse & Platform Engineering Design, build and maintain the Snowflake-based Data Lakehouse supporting Enhanced Data Architecture.
- Implement scalable data architecture across raw, processed and curated data layers.
- Develop reusable engineering patterns for ingestion, transformation and publishing of datasets.
- Support structured, semi-structured and unstructured data from enterprise systems, APIs, databases and file-based sources.
- Ensure the architecture can support downstream BI, analytics, AI/ML and data-sharing use cases.
- Data Pipeline Development
- Design, develop and maintain automated ETL/ELT data pipelines.
- Build reusable ingestion pipelines to reduce one-off development for new data sources.
- Develop transformation workflows to clean, standardise, validate and curate enterprise datasets.
- Implement error handling, logging, monitoring, reconciliation and pipeline recovery processes.
- Optimise pipelines for scalability, maintainability and performance.
- SQL Snowflake Engineering
- Develop complex SQL scripts, stored procedures, views and transformation logic.
- Design and optimise data models and schemas for analytical workloads.
- Configure and manage appropriate Snowflake capabilities for ingestion, transformation, monitoring and secure data consumption.
- Monitor query and compute performance and optimise Snowflake workloads and cost.
- Implement Snowflake security controls including role-based access control (RBAC) and appropriate data-access policies.
- Data Quality, Governance & Security
- Implement automated data-quality checks, validation rules and reconciliation controls.
- Support metadata management, data cataloguing and data lineage.
- Work with data owners and stakeholders to establish trusted and governed datasets.
- Implement appropriate data classification and access-control requirements.
- Ensure engineering solutions comply with applicable government security, governance and data-management requirements.