To be part of an existing team to create, maintain, support and improve upon existing internal production applications to speed up delivery of products to our customers.
Requirements
- Develop technical solutions that help achieve business and customer goals.
- Provide clear feedback to technical and project teams upon encountering technological limitations.
- Work with project management to assist in work breakdowns and estimates as required.
- Ensure documentation is created and maintained with a view to enabling ongoing development support and improvement.
- Ensure ongoing code quality, data quality checking, error reporting and correction occurs.
- 2+ years of E2E In-depth Databricks experience on a real project(s) that have gone to production
- Databricks Genie, AI/BI Dashboards and Analytics reporting.
- Databricks pipelines, warehouse, jobs.
- Databricks Platform set up and admin, cost reduction.
- Databricks ELT, CDC, Streaming - general ETL and ELT and ETL/ELT scaling.
- Databricks Ingestion tools, e.g., SQL Server Connect
- Development of Databricks applications, ‘App’ and ‘One’ experience.
- Strong experience with Delta Lake, Lakehouse and Medallion architectures, Data architecture
- Meta data catalogs and data governance - Databricks Unity catalog.
- Strong data modelling/data expertise.
- Understanding of AI and Ml.
- General computing experience, AWS, S3, CI/CD, Git, SDLCs, Agile/Iterative.
- Excellent communication and time management skills
- Strong analytical and problem-solving skills
- Be flexible/adaptable and can embrace change
- DB/Spark performance and optimization.
- Strong Apache Spark and pyspark - created Spark clusters, Spark (and Kafka) Streaming, Data pipelines/orchestration.
- Strong Python and Dash development experience.
- Advance SQL, with materialized views, Common Table Expressions.
- Statistics, Machine Learning, data mining, Experimental design.
- MLFlow, AI training/testing/evaluation and its orchestration.
- GenAI, LLMs, Prompt Engineering, RAG, Fine Tuning, Agents.
- AI ways of working, e.g. CRISP-DM, TDSP etc.