Senior Data Engineer

AWS Senior Data Engineer - AI

Key Responsibilities

  • Design, build, and optimise scalable ETL/ELT pipelines for batch and streaming data.
  • Develop and orchestrate workflows using AWS Glue and Apache Airflow.
  • Implement and manage serverless data processing solutions using AWS Lambda and other AWS native services.
  • Write and optimise complex SQL queries for data transformation, modelling, and reporting.
  • Drive CI/CD implementation for data workflows, ensuring reliability and repeatability.
  • Partner with stakeholders to design data models and ensure best practices for data architecture.
  • Monitor, troubleshoot, and enhance pipeline performance, focusing on cost management and optimisation.
  • Collaborate with cross-functional teams to support real-time analytics and business intelligence initiatives.
  • Stay up to date with emerging data engineering practices and AWS innovations.
Required Skills & Experience
  • 8+ years of professional experience in Data Engineering and related area.
  • Expertise with AWS cloud services (Glue, Lambda, S3, Redshift, Kafka, Kinesis, etc.).
  • Proven experience with workflow orchestration tools (Airflow, Step Functions, or equivalent).
  • Solid knowledge of SQL and experience in relational and non-relational databases.
  • Hands-on experience designing and maintaining data pipelines for both batch and streaming data.
  • Proficiency in CI/CD concepts and implementation (GitHub Actions, CodePipeline, or equivalent).
  • Strong understanding of data modelling concepts, performance tuning, and cost optimisation in cloud environments.
  • Proficiency in at least one programming language such as Python, Java, or Scala.
  • Excellent problem-solving skills and the ability to work independently as well as in a team.
Nice to Have
  • Experience with real-time data streaming tools (Kafka, Kinesis Data Streams, or Flink).
  • Familiarity with containerisation and orchestration (Docker, Kubernetes).
  • Exposure to machine learning pipelines or supporting data science workloads.
  • Understanding of data governance, security, and compliance in cloud-based systems.
  • Aviation or Airlines experience a bonus