Databricks Architect

Seeking a Databricks Architect with 8+ years of experience in developing and implementing ETL pipelines using Databricks on cloud, with some experience in the insurance domain. Responsibilities include leading data engineering projects, integrating data across different systems, and utilizing Spark Structured Streaming for real-time data processing.

Requirements

• Develop and optimize ETL pipelines from various data sources using Databricks on cloud
• Implement standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
• Continuously improve systems through performance enhancements and cost reductions in compute and storage
• Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
• Lead Data Engineering Projects to manage and implement data-driven communication systems
• Experience with Scrum and Agile Methodologies
• Strong verbal and written communication skills to manage client discussions
• Ability to work independently with some level of ambiguity and juggle multiple demands