ML Systems Engineer – Training & Inference
ZenteiQ.aiBengaluru, IndiaOn-Site
Needs a free Superjobs account. You'll finish on ZenteiQ.ai's site.
Skill score
Summary
This engineer builds and runs the infrastructure that lets researchers train, evaluate, and serve large foundation models reliably on Cloud TPUs, owning distributed training, inference, observability, and accelerator capacity. It suits someone with strong distributed-systems and production Python experience who wants to work at the intersection of ML research and platform engineering.
- Function
- Data Science
- Estimated salary
- Est. ₹9.5–14 LPA
- Estimated for a mid-level role, ~3 yrs experience (typical for the level), India. Not published by the employer.
- Work mode and location
- On-Site, Bengaluru, India
Job description
Overview
You will build the systems that let our researchers train, evaluate, and serve large foundation models reliably at scale. This role sits at the intersection of model research and infrastructure, with a focus on accelerator-based (TPU) training and inference, performance, reproducibility, and researcher velocity. You will own the paved path that turns expensive, long-running model runs into a repeatable, observable, and cost-efficient process.
Roles and responsibilities
- Build and improve the paved path for distributed training, evaluation, experiment tracking, checkpoint management, model release, and inference on Cloud TPUs.
- Operate long-running ML workloads with strong observability, failure detection, automated recovery, and practical operational tooling.
- Profile and remove bottlenecks across compute, HBM and host memory, data input, networking and collectives, XLA compilation, checkpointing, and serving.
- Build validation, representative-scale testing, CI/CD, reproducibility, and lineage systems that catch problems before expensive model runs or production releases.
- Create reusable APIs, abstractions, and self-service tooling that help researchers move quickly while preserving useful low-level controls.
- Own accelerator capacity workflows, including TPU provisioning, quotas, reservations, priorities, topology-aware placement, utilization, and cost efficiency.
- Partner with research teams to debug model and systems failures, translate recurring pain points into durable platform improvements and define operational standards.
Skills required
Top skills
- Distributed training
- Cloud TPU
- Kubernetes
- GKE
All skills (8)
- Production Python
- Observability
- Performance engineering
- Incident response
- JAX
- PyTorch/XLA
- LLM serving
- Terraform
Qualifications
- Strong software engineering skills and experience owning systems used by researchers or engineers in production or research-critical environments.
- Hands-on experience with ML training or inference infrastructure at meaningful scale, including large accelerator or distributed compute workloads.
- Strong distributed-systems fundamentals and experience with Kubernetes, GKE, or comparable orchestration for long-running compute jobs.
- Ability to debug across model code, data pipelines, runtimes, cluster services, storage, networking, and accelerator behaviour.
- Strong observability, reliability, and performance-engineering fundamentals, including incident response and root-cause analysis.
- Production-quality Python and the ability to build durable platform abstractions rather than one-off scripts.
- High ownership, pragmatic judgment, and clear collaboration with fast-moving research teams.
Nice to have
- Experience operating large Cloud TPU clusters, including TPU VMs or Pods, multislice jobs, topology, quotas, scheduling, and failure recovery.
- Experience with JAX, PyTorch/XLA, TensorFlow, XLA/HLO, PJRT, MaxText, Pallas, or another large-scale training stack.
- Experience serving LLMs on TPUs, including batching, partitioning, compilation caching, autoscaling, latency, and throughput optimization.
- Familiarity with GCP/GKE, Terraform, Vertex AI, XPK, Ray, Argo, Airflow, MLflow, Weights
& Biases, or comparable internal platforms.
- Experience with model registries, distributed checkpointing, data and artifact lineage, and large artifact distribution.
- Proficiency in a systems language such as Go, Rust, C++, or Java.
Company info
About ZenteiQ.ai
ZenteiQ builds scientific foundation models grounded in physics, engineered for real-world industry. The sources do not name the customers or users of its models or products.