Skip to content

ML Systems Engineer – Training & Inference

ZenteiQ.aiBengaluru, IndiaOn-Site

Estimated salary, not published by the employer: Est. ₹9.5–14 LPA

Needs a free Superjobs account. You'll finish on ZenteiQ.ai's site.

Skill score

Summary

This engineer builds and runs the infrastructure that lets researchers train, evaluate, and serve large foundation models reliably on Cloud TPUs, owning distributed training, inference, observability, and accelerator capacity. It suits someone with strong distributed-systems and production Python experience who wants to work at the intersection of ML research and platform engineering.

Function
Data Science
Estimated salary
Est. ₹9.5–14 LPA
Estimated for a mid-level role, ~3 yrs experience (typical for the level), India. Not published by the employer.
Work mode and location
On-Site, Bengaluru, India

Job description

Overview

You will build the systems that let our researchers train, evaluate, and serve large foundation models reliably at scale. This role sits at the intersection of model research and infrastructure, with a focus on accelerator-based (TPU) training and inference, performance, reproducibility, and researcher velocity. You will own the paved path that turns expensive, long-running model runs into a repeatable, observable, and cost-efficient process.

Roles and responsibilities

  • Build and improve the paved path for distributed training, evaluation, experiment tracking, checkpoint management, model release, and inference on Cloud TPUs.
  • Operate long-running ML workloads with strong observability, failure detection, automated recovery, and practical operational tooling.
  • Profile and remove bottlenecks across compute, HBM and host memory, data input, networking and collectives, XLA compilation, checkpointing, and serving.
  • Build validation, representative-scale testing, CI/CD, reproducibility, and lineage systems that catch problems before expensive model runs or production releases.
  • Create reusable APIs, abstractions, and self-service tooling that help researchers move quickly while preserving useful low-level controls.
  • Own accelerator capacity workflows, including TPU provisioning, quotas, reservations, priorities, topology-aware placement, utilization, and cost efficiency.
  • Partner with research teams to debug model and systems failures, translate recurring pain points into durable platform improvements and define operational standards.

Skills required

Top skills

  • Distributed training
  • Cloud TPU
  • Kubernetes
  • GKE

All skills (8)

  • Production Python
  • Observability
  • Performance engineering
  • Incident response
  • JAX
  • PyTorch/XLA
  • LLM serving
  • Terraform

Qualifications

  • Strong software engineering skills and experience owning systems used by researchers or engineers in production or research-critical environments.
  • Hands-on experience with ML training or inference infrastructure at meaningful scale, including large accelerator or distributed compute workloads.
  • Strong distributed-systems fundamentals and experience with Kubernetes, GKE, or comparable orchestration for long-running compute jobs.
  • Ability to debug across model code, data pipelines, runtimes, cluster services, storage, networking, and accelerator behaviour.
  • Strong observability, reliability, and performance-engineering fundamentals, including incident response and root-cause analysis.
  • Production-quality Python and the ability to build durable platform abstractions rather than one-off scripts.
  • High ownership, pragmatic judgment, and clear collaboration with fast-moving research teams.

Nice to have

  • Experience operating large Cloud TPU clusters, including TPU VMs or Pods, multislice jobs, topology, quotas, scheduling, and failure recovery.
  • Experience with JAX, PyTorch/XLA, TensorFlow, XLA/HLO, PJRT, MaxText, Pallas, or another large-scale training stack.
  • Experience serving LLMs on TPUs, including batching, partitioning, compilation caching, autoscaling, latency, and throughput optimization.
  • Familiarity with GCP/GKE, Terraform, Vertex AI, XPK, Ray, Argo, Airflow, MLflow, Weights

& Biases, or comparable internal platforms.

  • Experience with model registries, distributed checkpointing, data and artifact lineage, and large artifact distribution.
  • Proficiency in a systems language such as Go, Rust, C++, or Java.

Company info

ZenteiQ builds scientific foundation models grounded in physics, engineered for real-world industry. The sources do not name the customers or users of its models or products.

More jobs at ZenteiQ.ai