Data Platform Engineer

Kestra Holdings is seeking a seasoned Databricks Data Engineer to design and optimize large-scale data pipelines on the Databricks Lakehouse platform. The role involves modernizing cloud-based data ecosystems, enabling secure, governed data solutions, and working with financial or regulated datasets. The successful candidate will have expertise in Azure cloud services, SQL, Python, PySpark, and ETL/ELT frameworks, as well as strong compliance and governance practices.

Requirements

• Design and optimize large-scale data pipelines on the Databricks Lakehouse platform
• Modernize the Azure-based data ecosystem
• Utilize Apache Airflow and similar tools for orchestration and workflow automation
• Work with financial or regulated datasets, applying strong compliance and governance practices
• Develop and optimize ETL/ELT pipelines using Python, PySpark, Spark SQL, and Databricks notebooks
• Design and optimize Delta Lake data models for reliability, performance, and scalability
• Implement and manage Unity Catalog for RBAC, lineage, governance, and secure data sharing
• Build reusable frameworks using Databricks Workflows, Repos, and Delta Live Tables
• Create scalable ingestion pipelines for APIs, databases, files, streaming sources, and MDM systems
• Automate API ingestion and workflows using Python and REST APIs
• Support data governance, lineage, cataloging, and metadata initiatives
• Enable downstream consumption for BI, data science, and application workloads
• Write optimized SQL/T-SQL queries, stored procedures, and curated datasets for reporting
• Automate deployments, DevOps workflows, testing pipelines, and workspace configuration

Benefits

• Competitive pay and benefits with a large employer (over 1600 employees nationwide)
• 401(k), health insurance, and a competitive benefits package
• Work in a supportive, collaborative environment committed to professional excellence
• Help clients navigate meaningful financial decisions with confidence
• Opportunities for training, development, and long-term growth within the firm
• Tuition reimbursement for qualified expenses