Software Engineer, Data Infrastructure

Companies want to train their own large models on their own data. The current industry standard is to train on a random sample of your data, which is inefficient at best and actively harmful to model quality at worst. There is compelling research showing that smarter data selection can train better models faster—we know because we did much of this research.

Requirements

• Have meaningful experience with leading and building production data systems to deliver on major product initiatives.
• Have built and managed highly scalable data processing solutions (e.g. Spark, Flink), data lakes or warehouses (e.g. Snowflake, Hive), authored queries (SQL), distributed storage systems (e.g., HDFS, S3), used workflow management (e.g. Airflow, Dagster), and have experience maintaining the infra that supports these.
• Proficiency in at least one programming language commonly used within Data Engineering, such as Python, Scala, or Java.
• Expertise with any of ETL schedulers such as Airflow, Dagster, or similar frameworks.
• Experience maintaining a high quality bar for design, correctness, and testing.
• Take pride in building and operating scalable, reliable, secure systems
• Have a humble attitude, an eagerness to help your colleagues, and a desire to do whatever it takes to make the team succeed
• Own problems end-to-end, and are willing to pick up whatever knowledge you're missing to get the job done
• Have experience being the technical lead of a Data Engineering / Platform / Infrastructure Team.
• Experience building ML/DL systems and/or data infrastructure that feeds into training large ML models

Benefits

• 100% covered health benefits (medical, vision, and dental).
• 401(k) plan with a generous 4% company match.
• Unlimited paid time off (PTO) policy.
• Annual $2,000 wellness stipend.
• Annual $1,000 learning and development stipend.
• Daily lunches and snacks are provided in our office!
• Relocation assistance for employees moving to the Bay Area.