Senior AI/ML Engineer — LLM & Agent Stack (Customer Facing)
TrueFoundryNew York, United StatesOn-Site
Needs a free Superjobs account. You'll finish on TrueFoundry's site.
Skill score
Summary
The Senior AI/ML Engineer designs and owns core components of TrueFoundry's LLM and agent stack, including agent orchestration, model routing, observability, and guardrails for enterprise customers. The role suits an engineer with deep distributed systems and LLM production experience who is comfortable working directly with customers and writing code in their environments.
- Function
- Data Science
- Estimated salary
- Est. $155k–232k/yr
- Estimated for a senior role, ~5 yrs experience, United States. Not published by the employer.
- Work mode and location
- On-Site, New York, United States
- Experience
- 5–10 yrs
Job description
Overview
We are looking for a Senior AI/ML Engineer—LLM & Agent Stack (Customer-Facing) to join the team.
You’ll design and own core components that enable enterprise customers to run production agentic AI safely and efficiently on TrueFoundry. This includes building robust orchestration for multi-step agents (graph/stateful workflows), model/routing logic, observability and policy enforcement (cost, data residency, rate limiting), and integrating upstream tooling like LangGraph, LangChain, vector stores, and specialized LLM runtimes. This is a customer-facing role where you'll work closely with TrueFoundry's enterprise customers. You may also need to write code in their environments.
Roles and responsibilities
- Architect and implement scalable agent orchestration patterns (graph-based executors, state management, multi-agent coordination) for production workloads.
- Own critical integrations: model adapters, LLM gateway hooks, vector DBs, tools & external APIs, and the platform’s LLMops flows.
- Build and improve tracing, benchmarking and observability for LLMs and agents — token/cost accounting, latency p95, throughput, and correctness checks.
- Drive design for safety/guardrails: moderation hooks, human-in-the-loop checkpoints, replayable audit trails and policy enforcement.
- Mentor junior engineers, run design reviews, and improve engineering practices (testing, CI/CD, chaos testing for agents).
- Work directly with strategic customers to prototype complex agentic solutions and translate them into product features.
Skills required
Top skills
- Distributed systems
- LLM production deployment
- RAG
- Embeddings pipelines
All skills (12)
- LangGraph
- LangChain
- Kubernetes
- Container orchestration
- Service mesh
- Performance tuning
- Observability
- Cost controls
- Policy enforcement
- Open-source LLM tools
- Data residency compliance
- Model governance
Qualifications
- 5–10 years of software engineering with substantial experience building distributed systems, infra, or ML platforms.
- Deep practical experience integrating and deploying LLMs in production (RAG, retrieval, embeddings pipelines).
- Hands-on experience with agent orchestration frameworks (LangGraph / LangChain or custom agent runtimes) and stateful workflow design.
- Strong systems knowledge: Kubernetes, container orchestration, service meshes, and performance tuning.
- Proven track record building observability, cost controls, and policy enforcement for production services.
Nice to have
- Experience building or contributing to open-source LLM orchestration tools (LangGraph, LangChain, or similar).
- Familiarity with enterprise constraints: on-prem/cloud hybrid deployments, data residency, compliance requirements.
- Background in security, privacy, or model governance for LLMs.
- Demonstrated leadership in cross-functional projects and direct customer engagement.
Other details
Traits we are looking for: ownership, ability to execute, hustle and thinking out of the box, data-driven decision making, be comfortable with more unknowns than knowns.
Benefits
- Comprehensive health insurance for you and your family, including medical, dental, and vision coverage.
- 401(k) retirement plan.
- Flexible hybrid work, 2 days a week in the office (Tuesday & Wednesday), with flexibility around the schedule.
- Lunch and snacks are on us whenever you come into the office.
- Work from another TrueFoundry office for up to 1 month each year, whether that's our London or India office, if you'd like a change of scenery.
Company info
About TrueFoundry
TrueFoundry provides an enterprise-grade AI Gateway, comprising an LLM Gateway, MCP Gateway, and Agent Gateway, that lets organizations connect, observe, and govern agentic AI applications across multiple providers from a single control plane. Its product is aimed at enterprises building and running agentic AI applications.