Data Engineer

Data Engineer responsible for designing, developing, and maintaining ETL/ELT pipelines using Apache Spark and AWS Glue, processing large volumes of financial information in cloud environments, and implementing data transformation logic and conversion of accounting data.

Requirements

• Design, develop, and maintain ETL/ELT pipelines using Apache Spark and AWS Glue.
• Process, transform, and load large volumes of financial information in cloud environments (AWS).
• Implement transformation logic and conversion of accounting data (account mapping, conversion rules, and data integrity checks).
• Manage data catalogs in AWS Glue Data Catalog.
• Optimize queries and processing over S3, Redshift, and RDS.
• Ensure data quality, traceability, and lineage at all stages of the pipeline.
• Develop Python scripts for data automation, cleaning, and validation.