Senior AI Software Engineer - Model Evaluation (f/m/d)

Aleph Alpha Research is seeking a Senior AI Software Engineer to work on model evaluation. The role involves designing and implementing evaluation suites, building evaluation infrastructure, and correlating signals to predict downstream performance. The ideal candidate has experience with LLM evaluation, benchmark design, and statistical methods for evaluation and experiment design.

Requirements

• Experience with LLM evaluation, benchmark design, evaluation dataset curation, and experimental design.
• Familiarity with statistical methods for evaluation and experiment design.
• Track record of shipping impactful technical work - whether that's research, infrastructure, or both.
• Strong Python skills and comfort with ML tooling (PyTorch, evaluation frameworks, distributed systems).
• Ability to reason about what an evaluation measures and whether it matters - not just run benchmarks, but understand them.
• Ownership mentality: you see problems through from diagnosis to solution to deployment.
• Willingness to relocate to Heidelberg or travel regularly (potentially weekly).

Benefits

• 30 days of paid vacation
• Access to a variety of fitness & wellness offerings via Wellhub
• Mental health support through nilo.health
• Substantially subsidized company pension plan for your future security
• Subsidized Germany-wide transportation ticket
• Budget for additional technical equipment
• Flexible working hours for better work-life balance and hybrid working model
• Virtual Stock Option Plan