Skip to content

Research Engineer

EntelligenceSan Francisco, CA, USAOn-Site

Estimated salary, not published by the employer: Est. $120k–180k/yr

Needs a free Superjobs account. You'll finish on Entelligence's site.

Skill score

Summary

The research engineer improves a coding-agent optimization engine by mining agent sessions and PR review comments for failures, attributing them to harness problems, and proving that learned rules or judge checks reduce repeat findings. They also run the benchmark harness and model cost and outcome modeling, which suits someone with production ML or LLM experience and strong experiment design skills.

Function
Development
Estimated salary
Est. $120k–180k/yr
Estimated for a mid-level role, ~3 yrs experience, United States. Not published by the employer.
Work mode and location
On-Site, San Francisco, CA, USA
Experience
3+ yrs

Job description

Overview

We want a research engineer to make it measurably better at both.

Roles and responsibilities

  • Failure attribution. Given a session transcript and the review comments on its PR, find where the harness went wrong: missing context, a bad tool loop, a skipped check, the wrong model.
  • Prevention that measurably works. Decide which lessons become rules, cards or judge checks. Prove with held-out data that they cut repeat findings without flooding agents with noise.
  • Evaluation. Own the benchmark harness of record. Replay real agentic tasks headlessly against competing harness configs and routing policies. Measure quality vs. cost honestly.
  • Engineering world model. Build the data layer connecting sessions, PRs, projects, errors and incidents. Use it to catch issues before they happen in future sessions.
  • Cost and outcome modeling. Turn measured usage, cache shares, latency and review outcomes into continuously refreshed model and harness parameters.

Skills required

Top skills

  • Python
  • LLM APIs
  • Agent tool-use loops
  • Experiment design

All skills (8)

  • Held-out evaluation
  • Statistical confidence intervals
  • Agent evaluation
  • LLM-as-judge
  • Model routing
  • Prompt caching
  • SWE-bench
  • Failure analysis

Qualifications

  • 3+ years building ML or LLM systems that run in production, or equivalent research experience with shipped code.
  • Strong Python. Comfortable reading a large production codebase.
  • Experiment design you can defend: held-out sets, variance, confidence intervals, leakage checks.
  • Hands-on experience with LLM APIs, agents or tool-use loops, and their failure modes.

Nice to have

  • Agent evaluation, failure analysis, LLM-as-judge or automated code review.
  • Routing, cascades, bandits or learning-to-defer.
  • LLM inference economics: prompt caching, batching, token pricing.
  • Built or maintained public evals (SWE-bench, Terminal-Bench style harnesses).
  • Published research or well-known open-source work in ML systems or developer tools.

Other details

What the engine does today

  • Failure mining. Captures coding-agent sessions and links them to the PRs they produced. Mines transcripts and review comments for errors, struggles and recurring root causes.
  • Learning loop. Turns review findings into root-cause patterns and tested rules. Delivers them to the next agent session as guidance cards through CLI hooks.
  • Pre-PR judge. A reviewer model checks an agent's change before the PR opens. The harness reports back what happened in each session.
  • Cost-aware routing. Picks a difficulty tier per instruction turn and weights models by expected cost per solved task. Prompt-cache affinity, failover and per-org constraints built in.

Compensation

  • Base salary: $170,000 to $210,000, plus meaningful equity. Final offer depends on level and experience.

Benefits

  • Healthcare.
  • Generous compute budget for experiments.

Company info

Entelligence keeps production stable. Code review that remembers every incident you've had. Spend tracking for every AI agent on the team. Shared memory for coding agents. 100+ teams use it, including LangChain, DevRev, Composio and AngelList.

More jobs at Entelligence