Skip to content

Member of Technical Staff, Evals

MagicSan Francisco, RemoteRemote

Estimated salary, not published by the employer: Est. $215k–323k/yr

Needs a free Superjobs account. You'll finish on Magic's Ashby page.

Skill score

Summary

The Member of Technical Staff on Evals designs new benchmarks and evaluation methods, runs data ablations, assesses dataset quality, and checks whether measurements truly reflect model performance. This suits an engineer with ML and evaluation experience who is skeptical of unreproducible results and cares about measurement rigor.

Function
Development
Estimated salary
Est. $215k–323k/yr
Estimated for a staff-level role, ~9 yrs experience (typical for the level), United States. Not published by the employer.
Work mode and location
Remote, San Francisco, Remote
Employment type
Full-time

Job description

Overview

The Evals team measures the performance and capabilities of internal and external models, helping teams across Magic make informed decisions. The team supports pretraining, RL, data, inference, and product, and sits on the critical path of many of the company's most important decisions.

As a Member of Technical Staff on Evals, you will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether our measurements accurately reflect model performance.

Roles and responsibilities

  • Design and validate new evaluations and benchmarks across model training and deployment
  • Investigate model behavior and identify gaps in existing evaluations
  • Measure dataset quality and identify opportunities to improve training data
  • Improve evaluation correctness, reproducibility, and reliability
  • Audit and improve upon public benchmarks, research papers, and open-source implementations
  • Define metrics that accurately reflect model quality
  • Build tools that make evaluation results accessible and useful across Magic

Skills required

Top skills

  • Model evaluation
  • Benchmark design
  • Machine learning systems
  • Programming

All skills (4)

  • Post-training techniques
  • Experimental analysis
  • Dataset quality assessment
  • Data ablations

Qualifications

  • Experience developing model evaluations, benchmarks, or ML experiments
  • Strong programming skills and experience working with machine learning systems
  • Track record of owning technical projects end-to-end
  • Strong understanding of post-training techniques and their impact on model performance
  • Skepticism toward results that cannot be reproduced, validated, or explained
  • Experience analyzing model performance and interpreting experimental results
  • Comfortable navigating ambiguity and determining whether a measurement actually captures the behavior it claims to measure
  • Commitment to raising the standard of rigor in AI evaluations

Other details

Sweating the details matters on this team. Many benchmarks, papers, and open-source evaluation frameworks contain subtle bugs or flawed assumptions that lead to misleading conclusions. We care deeply about correctness, reproducibility, and measurement quality.

Compensation

  • Annual salary range between $225K - $550K depending on experience
  • Equity is a significant part of total compensation, in addition to salary
  • 401(k) plan with 6% salary matching
  • Generous health, dental, and vision insurance for you and your dependents
  • Unlimited paid time off
  • Visa sponsorship and relocation support for candidates moving to San Francisco

Benefits

  • A small, fast-moving, highly collaborative team working on frontier AI systems

Company info

Magic’s mission is to build safe AGI that accelerates humanity’s progress on the world’s most important problems. We believe the most promising path to safe AGI lies in automating research and code generation to improve models and solve alignment more reliably than humans can alone. Our approach combines frontier-scale pretraining,

More jobs at Magic