Tech Lead Scala/Spark - Mississauga - Canada
PhotonMississauga, Ontario, Canada
Posted February 6, 2025
Full Time
About the Company
Photon is a global leader in AI and digital solutions, accelerating AI adoption and enabling Fortune 100 clients to stay agile and future-ready. With expertise in IoT, API management, big data analytics, and more, Photon combines AI management, digital innovation, and engineering excellence to drive lasting transformation for F500 clients across billions of touchpoints daily. Employing thousands worldwide, Photon's comprehensive suite of services positions them as a distinctive player in the converging digital and AI landscape.
Job Description
The role involves developing and deploying data processing applications using Apache Spark and Scala, optimizing Spark applications for large-scale data, and collaborating with data scientists and business analysts. Experience with Cloudera Hadoop and related technologies is essential. The focus is on ensuring data integrity, accuracy, and security.
Requirements
- Minimum 10+ years of experience in application development.
- Strong hands-on experience in Apache Spark, Scala, and Spark SQL for distributed data processing.
- Hands-on experience with Cloudera Hadoop components like HDFS, Hive, Impala, HBase, Kafka, and Sqoop.
- Familiarity with other Big Data technologies: Apache Kafka, Flume, Oozie, and Nifi.
- Experience building and optimizing ETL pipelines using Spark.
- Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL.
- Knowledge of data warehousing concepts, dimensional modeling, and data lakes.
- Ability to troubleshoot and optimize Spark and Cloudera platform performance.
- Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).