Sr. Principal Data Processing Software Engineer-OSS
DataPelagoMountain View, California, United States • Hyderabad, Telangana, India
Posted September 25, 2024
Full Time
About the Company
DataPelago develops DataPelago Nucleus, a universal data processing engine that accelerates data workloads across diverse hardware and query engines, enabling AI systems to scale efficiently. By optimizing price/performance, the platform unlocks value from all data types, addressing the bottleneck that limits AI innovation. The company’s focus on accelerated computing and cross‑platform compatibility sets it apart in the AI infrastructure market. Backed by leading venture partners such as Qualcomm Ventures and Eclipse, DataPelago is positioned to drive the next wave of AI‑powered solutions.
Job Description
DataPelago is a company at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are looking for a Senior/Principal Data Processing Engineer - OSS to join our engineering team and shape the future of accelerated data processing.
Requirements
- Minimum 7+ years of relevant experience or Master's degree in Computer Science or related field
- 3+ years of deep technical experience in developing core components of Apache Spark, Apache Flink, or equivalent platforms
- 3+ years of experience in instrumenting, analyzing, and optimizing the performance of data processing engine components
- Sound knowledge of the architecture and internal operation of one or more of Apache Spark, Apache Flink, Presto/Trino
- Demonstrated experience in the design, development, and successful release of high-performance data processing engines
- Exceptional programming skills in C, C++, and Java
- Extensive development experience in Linux environments
- Strong analytical and problem-solving skills with a passion for performance optimization
Benefits
- Competitive compensation
- Stock options
- Comprehensive benefits package
- Leadership development opportunities