SSM Data Tech Lead

To be part of an existing team to create, maintain, support and improve upon existing internal production applications to speed up delivery of products to our customers.

Requirements

• Develop technical solutions that help achieve business and customer goals.
• Provide clear feedback to technical and project teams upon encountering technological limitations.
• Work with project management to assist in work breakdowns and estimates as required.
• Ensure documentation is created and maintained with a view to enabling ongoing development support and improvement.
• Ensure ongoing code quality, data quality checking, error reporting and correction occurs.
• 2+ years of E2E In-depth Databricks experience on a real project(s) that have gone to production
• Databricks Genie, AI/BI Dashboards and Analytics reporting.
• Databricks pipelines, warehouse, jobs.
• Databricks Platform set up and admin, cost reduction.
• Databricks ELT, CDC, Streaming - general ETL and ELT and ETL/ELT scaling.
• Databricks Ingestion tools, e.g., SQL Server Connect
• Development of Databricks applications, ‘App’ and ‘One’ experience.
• Strong experience with Delta Lake, Lakehouse and Medallion architectures, Data architecture
• Meta data catalogs and data governance - Databricks Unity catalog.
• Strong data modelling/data expertise.
• Understanding of AI and Ml.
• General computing experience, AWS, S3, CI/CD, Git, SDLCs, Agile/Iterative.
• Excellent communication and time management skills
• Strong analytical and problem-solving skills
• Be flexible/adaptable and can embrace change
• DB/Spark performance and optimization.
• Strong Apache Spark and pyspark - created Spark clusters, Spark (and Kafka) Streaming, Data pipelines/orchestration.
• Strong Python and Dash development experience.
• Advance SQL, with materialized views, Common Table Expressions.
• Statistics, Machine Learning, data mining, Experimental design.
• MLFlow, AI training/testing/evaluation and its orchestration.
• GenAI, LLMs, Prompt Engineering, RAG, Fine Tuning, Agents.
• AI ways of working, e.g. CRISP-DM, TDSP etc.