AI Research Engineer
dscoutSan Francisco, CAOn-Site
Needs a free Superjobs account. You'll finish on dscout's Greenhouse page.
Skill score
Summary
This AI Research Engineer leads research on whether AI works for real people over long time horizons, founding Dscout's first long-horizon benchmark and building rubric and QA science for live customer programs. It suits a research-first scientist with published work in evals or preference learning who can also build prototypes and publish.
- Function
- Development
- Estimated salary
- Est. $120k–180k/yr
- Estimated for a mid-level role, ~3 yrs experience (typical for the level), United States. Not published by the employer.
- Work mode and location
- On-Site, San Francisco, CA
Job description
Overview
This is a research-first role focused on questions where the right rubric, benchmark, or method does not yet exist. You will find the gaps, propose bold directions, and take projects from the first hypothesis through experiments and benchmark construction to publication. You will also build: the science has to run inside live customer programs.
Roles and responsibilities
Rubrics and QA science
- Design rubrics and test whether they measure what they claim: validity, reliability, and sensitivity.
- Measure how reliable human raters are, and when disagreement is noise versus real signal.
- Calibrate AI judges against real human judgment, and study when they drift away from it.
- Build quality signals that put evidence behind every data point we ship.
Long-horizon, real-user benchmarks
- Design Dscout’s first benchmark: real users, in real context, over long horizons. Multi-week tasks across tools, not one-shot prompts.
- Define the protocol, task design, scoring, and release. Work with our delivery team on fieldwork.
- Work out how to keep a living benchmark honest over time: drift, contamination, difficulty, and what counts as a meaningful change.
- Write the paper and own the public release.
Human judgment as training and eval signal
- Study when real-world human feedback improves a model and when it adds noise.
- Explore how preference and judgment data from representative people differ from expert data.
- Turn findings into methods our forward deployed engineers and delivery team can run in the field.
New research directions
- These areas are the core, not the whole list. You will have significant latitude to propose new programs, such as calibrating user simulations against real behavior, human-AI collaboration, and behavioral evaluation of agents.
- Co-author with outside labs and academics on studies that need real-user data. Help run our paper-reading group.
Skills required
Top skills
- LLM-as-judge evaluation
- rubric-based evaluation
- preference data
- applied statistics
All skills (9)
- inter-rater reliability
- Python
- experiment design
- technical writing
- agent evaluation
- psychometrics
- item response theory
- human-subjects research
- open-source benchmark contributions
Qualifications
- Published research in evals, preference learning, reward modeling, human-AI interaction, measurement, or agent evaluation.
- A PhD, a PhD in progress, or an MS with two or more strong research internships at labs or top industry groups. If you do not have advanced degrees but have a history of creative machine learning research work, this is just as desirable.
- Strong applied statistics: inter-rater reliability, confidence intervals, power, and a nose for when a metric misleads.
- Hands-on experience with LLM-as-judge, rubric-based evaluation, and preference data.
- Strong Python, and the ability to build research prototypes and evaluation pipelines other people run. You use AI coding tools as a real part of your workflow.
- Experience running experiments on messy human data, not only curated datasets.
- Clear technical writing, and the ability to explain your work to a lab researcher and to a product manager.
- Based in the Bay Area.
Nice to have
Nice to have: long-horizon or agentic evaluation work; psychometrics, item response theory, or human-subjects research; open-source benchmark or dataset contributions.
Other details
- A QA and rubric framework running on live customer programs, with documented validity.
- Benchmark v0 released, with a paper submitted to a top venue.
- At least one external team using or citing our work.
- Delivery colleagues using your methods without needing you in the room.
Hiring process
How we will hire: a conversation with the Head of AI Research; a research talk on your publication; a design exercise (how would you validate a rubric for a long-horizon task?); a conversation with the GM, forward deployed engineers, and delivery partners.
What we care about most. A strong publication record helps. We care most about whether you can find the question that matters, design a rigorous way to answer it, and move fast enough that the answer is still useful.
Compensation
Compensation: $250-400K OTE + Equity. Includes a built-in bonus and equity program, publication freedom, a conference budget, flexible PTO, 15 holidays, 12 weeks of paid parental leave, 401k match, and an education stipend.
This is an ideal list. If the role excites you and you do not check every box, we would still love to hear from you.
Of course, what is outlined above is an ideal set of expectations; however, business needs and other projects and tasks may shift, and additional tasks could be assigned at the discretion of your manager.
Benefits
It doesn't stop there. When you join the Dscout team, you will get:
- A strong and competitive compensation package with a built-in bonus and equity program.
- An incredible and progressive benefits package (for both you and your dependents) to support work/life balance, including flexible PTO, 15 company holidays, 12 weeks of paid parental leave, 401k match, and much more.
- An education stipend to support your growth & development, and a remote work stipend.
- A company that is open and transparent with our team. You will know what is happening and why it matters.
Company info
About dscout
dscout provides an AI-powered feedback and testing platform that lets teams gather input from real people to validate ideas and iterate on products. It is used by teams that want to build with more confidence based on direct user feedback.