Senior AI Quality Engineer

We're looking for a Senior AI Quality Engineer to join our AI Platform team and establish the standards for testing AI at Roofr. The role involves defining testing standards, building evaluation frameworks, and integrating quality gates into CI/CD pipelines. The ideal candidate has experience building eval frameworks for LLM-powered features and is familiar with LLM APIs and agent frameworks.

Requirements

• 5–8 years of software engineering or quality assurance experience
• Hands-on experience building eval frameworks for LLM-powered features
• Strong engineering fundamentals
• Experience integrating automated quality checks into CI/CD pipelines
• Familiarity with LLM APIs and agent frameworks (e.g. Anthropic Claude, OpenAI, or similar) and the specific quality challenges they introduce
• Experience designing human review workflows to complement automated evaluation
• Strong collaboration skills
• Comfort operating in an early-stage environment

Benefits

• Paid time off (1st week of employment mandatory PTO, 1 Friday off per month, company wide paid shutdown for the week between Christmas and New Years)
• Flexible time off
• 80% employer-paid benefits in the U.S. and 100% employer-paid premiums for Extended Healthcare and Dental in Canada
• RRSP/401k match
• Generous Parental Leave policy
• Home office setup stipend
• Internet and phone allowance
• Remote first culture
• Weekly Friday paydays