Data Engineer

KDA Consulting Inc. is seeking a Data Engineer to design, build, and maintain scalable data pipelines and architectures supporting mission-critical programs within the Intelligence Community (IC).

Requirements

• Design, develop, and maintain scalable ETL/ELT pipelines to ingest, transform, and load data from multiple sources
• Build and optimize data workflows supporting both batch and real-time processing
• Ensure high availability and reliability of data pipelines across enterprise systems
• Leverage AWS services such as S3, Glue, Lambda, Redshift, EMR, and Kinesis to support data engineering solutions
• Implement best practices for data storage, partitioning, and lifecycle management
• Optimize data storage and retrieval for performance, scalability, and cost efficiency
• Implement indexing, partitioning, and query optimization strategies
• Monitor and troubleshoot data pipeline performance issues
• Ensure data accuracy, consistency, and integrity across all data pipelines and systems
• Implement validation, monitoring, and error-handling mechanisms
• Support data governance practices and compliance with security and IC standards
• Work closely with data scientists, software engineers, and mission stakeholders to support analytics and operational needs
• Integrate data solutions with enterprise applications, APIs, and downstream analytics platforms
• Participate in Agile development processes and contribute to continuous improvement efforts