As a Data Engineer, you will be responsible for architecting, designing, implementing, enhancing, and maintaining highly scalable, available, secure, and elastic cloud-ready data solutions using cutting-edge technologies to support our predictive and prescriptive analytics needs. You will work with cross-functional teams to address supply chain data needs in demand and supply planning, replenishment, pricing, and optimization. You will design, build, and manage/monitor data pipelines, integrate analytics and data science output into business processes and workflows, and work with popular data discovery, analytics, and BI and AI tools in semantic-layer data discovery.
Requirements
- 3+ years in data engineering building production data pipelines (batch and/or streaming) with Spark on cloud.
- 2+ years hands-on Azure Databricks (PySpark/Scala, Spark SQL, Delta Lake) including Delta Lake operations, Unity Catalog, Databricks Jobs/Workflows or Delta Live Tables, Azure Data Factory for orchestration, integration with ADLS Gen2, Key Vault.
- Strong SQL across large datasets; performance tuning (joins, partitions, file sizing).
- Data quality at scale (e.g., Great Expectations/Deequ), monitoring and alerting; debug/backfill playbooks.
- DevOps for data: Git branching, code reviews, unit/integration testing (pytest/dbx), CI/CD (Azure DevOps/GitHub Actions).
- Infrastructure as Code (Terraform or Bicep) for Databricks workspaces, cluster policies, ADF, storage.
- Observability & cost control: Azure Monitor/Log Analytics; cluster sizing, autoscaling, Photon; cost/perf trade-offs.
- Proven experience collaborating with cross-functional stakeholders (analytics, data governance, product, security) to ship and support data products.
Benefits
- Generous medical, dental, vision and other great benefits
- Paid parental and medical leave programs
- 401(k) with a company match component and profit sharing
- 15 days of paid time off plus company holidays
- Hybrid work model with flexibility
- Tuition reimbursement and student loan repayment assistance