Abacus Insights is seeking a Sr. Data Engineer to join their dynamic and rapidly expanding Tech Ops division. The successful candidate will work with customers, data vendors, and internal engineering teams to design, implement, and optimize complex data integration solutions within a modern, large-scale cloud environment.
Requirements
- Bachelor's degree in Computer Science, Computer Engineering, or a closely related technical field.
- 5+ years of hands-on experience as a Data Engineer working with large-scale, distributed data processing systems in modern cloud environments.
- Working knowledge of U.S. healthcare data domainsâincluding claims, eligibility, and provider datasetsâand experience applying this knowledge to complex ingestion and transformation workflows.
- Expert-level proficiency in Python, SQL, and PySpark, including developing distributed data transformations and performance-optimized queries.
- Demonstrated experience designing, building, and operating production-grade ETL/ELT pipelines using Databricks, Airflow, or similar orchestration and workflow automation tools.
- Proven experience architecting or operating large-scale data platforms using dbt, Kafka, Delta Lake, and event-driven/streaming architectures, within a cloud-native data services or platform engineering environmentârequiring specialized knowledge of distributed systems, scalable data pipelines, and cloud-scale data processing.
- Experience working with structured and semi-structured data formats such as Parquet, ORC, JSON, and Avro, including schema evolution and optimization techniques.
- Strong working knowledge of AWS data ecosystem componentsâincluding S3, SQS, Lambda, Glue, IAMâor equivalent cloud technologies supporting high-volume data engineering workloads.
- Proficiency with Terraform, infrastructure-as-code methodologies, and modern CI/CD pipelines (e.g., GitLab) supporting automated deployment and versioning of data systems.
- Deep expertise in SQL and compute optimization strategies, including Z-Ordering, clustering, partitioning, pruning, and caching for large-scale analytical and operational workloads.
- Hands-on experience with major cloud data warehouse platforms such as Snowflake (preferred), BigQuery, or Redshift, including performance tuning and data modeling for analytical environments.
Benefits
- Generous Paid Time Off
- 401k Matching
- Retirement Plan
- Visa Sponsorship
- Four Day Work Week
- Generous Parental Leave
- Tuition Reimbursement
- Relocation Assistance
