
Research Engineer
EntelligenceSan Francisco, CA, USAOn-Site
Needs a free Superjobs account. You'll finish on Entelligence's site.
Skill score
Summary
The research engineer improves a coding-agent optimization engine by mining agent sessions and PR review comments for failures, attributing them to harness problems, and proving that learned rules or judge checks reduce repeat findings. They also run the benchmark harness and model cost and outcome modeling, which suits someone with production ML or LLM experience and strong experiment design skills.
- Function
- Development
- Estimated salary
- Est. $120k–180k/yr
- Estimated for a mid-level role, ~3 yrs experience, United States. Not published by the employer.
- Work mode and location
- On-Site, San Francisco, CA, USA
- Experience
- 3+ yrs
Job description
Overview
We want a research engineer to make it measurably better at both.
Roles and responsibilities
- Failure attribution. Given a session transcript and the review comments on its PR, find where the harness went wrong: missing context, a bad tool loop, a skipped check, the wrong model.
- Prevention that measurably works. Decide which lessons become rules, cards or judge checks. Prove with held-out data that they cut repeat findings without flooding agents with noise.
- Evaluation. Own the benchmark harness of record. Replay real agentic tasks headlessly against competing harness configs and routing policies. Measure quality vs. cost honestly.
- Engineering world model. Build the data layer connecting sessions, PRs, projects, errors and incidents. Use it to catch issues before they happen in future sessions.
- Cost and outcome modeling. Turn measured usage, cache shares, latency and review outcomes into continuously refreshed model and harness parameters.
Skills required
Top skills
- Python
- LLM APIs
- Agent tool-use loops
- Experiment design
All skills (8)
- Held-out evaluation
- Statistical confidence intervals
- Agent evaluation
- LLM-as-judge
- Model routing
- Prompt caching
- SWE-bench
- Failure analysis
Qualifications
- 3+ years building ML or LLM systems that run in production, or equivalent research experience with shipped code.
- Strong Python. Comfortable reading a large production codebase.
- Experiment design you can defend: held-out sets, variance, confidence intervals, leakage checks.
- Hands-on experience with LLM APIs, agents or tool-use loops, and their failure modes.
Nice to have
- Agent evaluation, failure analysis, LLM-as-judge or automated code review.
- Routing, cascades, bandits or learning-to-defer.
- LLM inference economics: prompt caching, batching, token pricing.
- Built or maintained public evals (SWE-bench, Terminal-Bench style harnesses).
- Published research or well-known open-source work in ML systems or developer tools.
Other details
What the engine does today
- Failure mining. Captures coding-agent sessions and links them to the PRs they produced. Mines transcripts and review comments for errors, struggles and recurring root causes.
- Learning loop. Turns review findings into root-cause patterns and tested rules. Delivers them to the next agent session as guidance cards through CLI hooks.
- Pre-PR judge. A reviewer model checks an agent's change before the PR opens. The harness reports back what happened in each session.
- Cost-aware routing. Picks a difficulty tier per instruction turn and weights models by expected cost per solved task. Prompt-cache affinity, failover and per-org constraints built in.
Compensation
- Base salary: $170,000 to $210,000, plus meaningful equity. Final offer depends on level and experience.
Benefits
- Healthcare.
- Generous compute budget for experiments.
Company info
About Entelligence
Entelligence keeps production stable. Code review that remembers every incident you've had. Spend tracking for every AI agent on the team. Shared memory for coding agents. 100+ teams use it, including LangChain, DevRev, Composio and AngelList.