We are seeking an experienced Senior Data Engineer to build and optimize scalable data platforms using Microsoft Fabric, Databricks (and/or Snowflake) to enable enterprise analytics, BI, and AI/Gen AI use cases.
Requirements
- Design, build, and maintain end-to-end data solutions on Microsoft Fabric, Databricks, including Pipelines, Notebooks, Lakehouse, Data Warehouse, and Semantic Models.
- Implement scalable data ingestion, transformation, and loading (ETL/ELT) using Fabric Pipelines and PySpark.
- Develop robust data models and schemas optimized for analytics, reporting, and AI-driven consumption.
- Engineer high-performance data solutions that meet requirements for throughput, scalability, quality, and security.
- Author efficient PySpark and SQL code for large-scale data transformation, data quality management, and business rule processing.
- Build reusable framework components for metadata-driven pipelines and automation.
- Optimize Lakehouse and Data Warehouse performance, including partitioning, indexing, Delta optimization, and compute tuning.
- Develop and maintain stored procedures and advanced SQL logic for operational workloads.
- Design and prepare feature-ready datasets for AI and GenAI applications.
- Collaborate with data scientists and ML engineers to productionize AI pipelines.
- Implement data governance and metadata practices required for responsible AI.
- Leverage Fabric, Databricks capabilities to orchestrate and monitor AI-related data workflows.
- Apply data governance, privacy, and security standards across all engineered assets.
- Implement monitoring, alerting, and observability best practices for pipelines and compute workloads.
- Drive data quality initiatives, including validation frameworks, profiling, and anomaly detection.
- Partner with analytics, BI, data science, and product teams to understand requirements and translate them into technical solutions.
- Mentor junior engineers and contribute to engineering standards and patterns.
- Participate in architecture reviews, technical design sessions, and roadmap planning.
- Develop and deliver dashboards and reports using Power BI and Tableau.
Benefits
- Generous Paid Time Off
- 401k Matching
- Retirement Plan
- Relocation Assistance
