Databricks Engineer

TheAILabb is seeking a skilled Databricks Engineer to design, develop, and optimize big data solutions using the Databricks Unified Analytics Platform. The ideal candidate will collaborate with cross-functional teams to drive data-driven insights and ensure scalable, high-performance data architectures.

Requirements

• Design, implement, and optimize big data pipelines in Databricks.
• Develop scalable ETL workflows to process large datasets.
• Leverage Apache Spark for distributed data processing and real-time analytics.
• Implement data governance, security policies, and compliance standards.
• Optimize data lakehouse architectures for performance and cost-efficiency.
• Collaborate with data scientists, analysts, and engineers to enable advanced AI/ML workflows.
• Monitor and troubleshoot Databricks clusters, jobs, and performance bottlenecks.
• Automate workflows using CI/CD pipelines and infrastructure-as-code practices.
• Ensure data integrity, quality, and reliability in all pipelines.

Benefits

• Competitive salary
• Strong insurance package
• Extensive learning and development resources