Abacus Insights is seeking a Senior Data Engineer to join its dynamic Tech Ops division. The successful candidate will work with customers, data vendors, and internal engineering teams to design, implement, and optimize complex data integration solutions in a modern cloud environment.
Requirements
- Bachelorâs degree in Computer Science, Computer Engineering, or a closely related technical field.
- 6 years of handsâon experience as a Data Engineer working with largeâscale, distributed data processing systems in modern cloud environments.
- Strong ability to communicate complex technical concepts clearly across both technical and nonâtechnical stakeholders.
- Expertâlevel proficiency in Python, SQL, and PySpark, including developing distributed data transformations and performanceâoptimized queries.
- Demonstrated experience designing, building, and operating productionâgrade ETL/ELT pipelines using Databricks, Airflow, or similar orchestration and workflow automation tools.
- Proven experience architecting or operating largeâscale data platforms using dbt, Kafka, Delta Lake, and eventâdriven/streaming architectures, within a cloudânative data services or platform engineering environmentârequiring specialized knowledge of distributed systems, scalable data pipelines, and cloudâscale data processing.
- Experience working with structured and semiâstructured data formats such as Parquet, ORC, JSON, and Avro, including schema evolution and optimization techniques.
- Strong working knowledge of AWS data ecosystem componentsâincluding S3, SQS, Lambda, Glue, IAMâor equivalent cloud technologies supporting highâvolume data engineering workloads.
- Proficiency with Terraform, infrastructureâasâcode methodologies, and modern CI/CD pipelines (e.g., GitLab) supporting automated deployment and versioning of data systems.
- Deep expertise in SQL and compute optimization strategies, including ZâOrdering, clustering, partitioning, pruning, and caching for largeâscale analytical and operational workloads.
- Handsâon experience with major cloud data warehouse platforms such as Snowflake (preferred), BigQuery, or Redshift, including performance tuning and data modeling for analytical environments.
Benefits
- Competitive Leave & Benefits
- Comprehensive health coverage
- Equity for every employee â share in our success
- Growth-focused environment â your development matters here
