SDE2 - API Platform
Sarvam AIBengaluruOn-Site
Needs a free Superjobs account. You'll finish on Sarvam AI's Ashby page.
Skill score
Summary
This is an IC backend role owning core parts of Sarvam's LLM API platform, covering streaming, routing and failover, metering and billing, reliability, and observability, while working in both Python and Go during the rewrite. It suits a backend engineer with 3+ years of production experience who can operate multi-tenant, low-latency services and is willing to learn Go.
- Function
- Development
- Estimated salary
- Est. ₹8–12 LPA
- Estimated for a mid-level role, ~3 yrs experience, India. Not published by the employer.
- Work mode and location
- On-Site, Bengaluru
- Experience
- 3+ yrs
- Employment type
- Full-time
Job description
Overview
Sarvam's model APIs (LLM, ASR, TTS, Vision, Translate) are how developers and enterprises ship products on our foundation models. Our OpenAI-compatible LLM APIs route every request across multiple model deployments, with streaming, failover, metering, billing, and rate-limiting all part of the same stack. Today it's FastAPI and Python; we're in the middle of rewriting it in Go.
We're hiring a Software Engineer (SDE 2) to own core parts of this platform. The features. The reliability. The performance. This is an IC role.
Roles and responsibilities
- Building and evolving the LLM API platform, from the moment a request hits the edge to the response that goes back out
- The Python→Go rewrite: building services in Go alongside the existing Python stack, without compromising reliability
- Streaming infrastructure: SSE for Chat Completions and Responses, long-lived connections, clean handling of cancellations and client disconnects
- Routing and failover: weighted provider chains, retries with backoff, and latency-driven admission control across model deployments
- Protocol compatibility: keeping the API faithful to the OpenAI spec across inference servers (vLLM, SGLang, Dynamo) that each behave slightly differently
- The commercial layer: metering, billing, and rate-limiting, engineered to the same bar as the model APIs themselves
- Performance and reliability for a multi-tenant platform serving long-context agentic and coding workloads
- Observability (logging, metrics, tracing), so the team can debug production confidently and quickly
- Integration and load testing: the harness that lets us ship fast without breaking customer-facing APIs
- Close partnership with the Inference and MLOps teams on production readiness for new model launches
Skills required
Top skills
- Python
- asyncio
- FastAPI
- Go
All skills (8)
- PostgreSQL
- Redis
- Kubernetes
- SSE or WebSockets streaming
- Rate limiting
- Observability (logs, metrics, tracing)
- Multi-tenant backend services
- vLLM / SGLang inference servers
Qualifications
- 3+ years building production backend systems
- Strong Python in production: asyncio, FastAPI or similar, and services you've built and operated under real load
- Working knowledge of Go, or a strong willingness to learn it: you'll work in both languages through the rewrite
- Experience building low-latency, multi-tenant backend services
- Hands-on experience with streaming systems: SSE or WebSockets, long-lived connections, backpressure
- Strong PostgreSQL and Redis fundamentals: schema design, query performance, caching, and rate limiting
- Comfort running services on Kubernetes in production
- An ownership mindset: you test before you ship, debug production with metrics, logs, and traces, and write the RCA afterwards
Nice to have
- Experience serving or integrating LLMs in production, especially OpenAI-compatible APIs
- Familiarity with inference servers (vLLM, SGLang, Dynamo) and latency metrics such as time to first token and inter-token latency
- Experience building metering, billing, or usage-based pricing infrastructure
- Time at an early- or growth-stage startup: comfort with ambiguity, speed, and ownership
Company info
About Sarvam AI
Sarvam AI is a sovereign AI platform for India, offering speech-to-text, text-to-speech, translation, and conversational agents across 22 Indian languages. Its products are used by enterprises and public institutions.