Software Engineer (Scheduler), London
Isomorphic LabsLondonOn-Site
Needs a free Superjobs account. You'll finish on Isomorphic Labs's Greenhouse page.
Skill score
Summary
The engineer will own and extend the systems that schedule compute for ML batch inference and training on Kubernetes, optimizing GPU and TPU utilization and building custom controllers. It suits a distributed systems and infrastructure engineer who is comfortable with Kubernetes internals and batch schedulers.
- Function
- Development
- Estimated salary
- Est. $73k–109k/yr
- Estimated for a mid-level role, ~3 yrs experience (typical for the level), United Kingdom. Not published by the employer.
- Work mode and location
- On-Site, London
Job description
Overview
You will join our newly formed Scheduling team to own and extend the systems that orchestrate compute allocation to complex machine learning workloads running on Kubernetes. You will be responsible for optimizing our compute footprint, ensuring that dynamic batch inference and large-scale training runs are scheduled reliably and efficiently.
We are looking for an engineer who operates comfortably at the intersection of distributed systems and modern ML infrastructure. You will design systems that support large-scale, topology-sensitive workloads, optimize batch queuing frameworks and write custom Kubernetes controllers. Your work will directly determine our ability to maximize hardware utilization and accelerate the pace of AI-driven drug discovery.
Roles and responsibilities
- Own the Scheduling Layer: Design, operate, and extend ML workload scheduling systems to handle dynamic batch inference and long-running training workloads.
- Optimize Compute Utilization: Drive the implementation of scheduling policies like gang scheduling, topology-aware placement, and preemption to ensure efficient use across our fleets of committed GPUs and TPUs.
- Partner with Research: Collaborate directly with ML researchers and the broader Infrastructure team to understand workload behavior, troubleshoot bottlenecks, and turn research requirements into platform capabilities.
- Ensure Reliability: Build systems with clear failure semantics. Participate in the operational lifecycle of the scheduling ecosystem, designing robust processes, runbooks, and SLOs to maintain high availability.
- Build Custom Infrastructure: Build and maintain custom Kubernetes controllers, operators, and CRDs that adapt the cluster orchestration layer to the specific requirements of our ML pipelines.
Skills required
Top skills
- Kubernetes
- Batch scheduling frameworks
- Kueue, Volcano or Slurm
- Custom K8s controllers
All skills (8)
- Cloud infrastructure
- Infrastructure as Code
- DevOps/SRE practices
- Python or Go
- Nvidia GPUs
- Google TPUs
- Gang scheduling
- GCP
Qualifications
- Scheduling Frameworks: Practical experience building or operating cluster schedulers, batch systems, or queuing frameworks (e.g., Kueue, Volcano, Slurm).
- Deep Kubernetes Expertise: Deep, hands-on Kubernetes experience, moving beyond standard usage to write and maintain custom K8s controllers, operators, or interacting directly with cluster orchestration APIs. Strong familiarity with K8s scheduler.
- Cloud Infrastructure Experience: Strong practical experience with cloud providers, focusing on compute provisioning, networking, and Infrastructure as Code to build scalable foundational environments.
- Adaptability: A solid foundation in DevOps/SRE best practices, with the ability to adapt these principles to the dynamic nature of AI accelerator infrastructure.
- Programming Proficiency: Strong programming skills, with high proficiency in Python or Go.
Nice to have
- Experience operating Kueue.
- Familiarity with ML infrastructure nuances: Nvidia GPUs, Google TPUs, gang scheduling, or collective networking.
- Previous experience in High-Performance Computing or maximizing utilization in environments with fixed/committed hardware constraints.
- Familiarity with GCP cloud compute infrastructure and Infrastructure as Code.
- Background in ML SWE or infrastructure SRE work supporting machine learning research activities.
Other details
Hybrid working
It’s hugely important for us to share knowledge and build strong relationships with each other, and we find it easier to do this if we spend time together in person. This is why we follow a hybrid model, and would require you to be able to come into the office 3 days a week (currently Tuesday, Wednesday, and one other day depending on which team you’re in). If you have additional needs that would prevent you from following this hybrid approach, we’d be happy to talk through these if you’re selected for an initial screening call.
Company info
About Isomorphic Labs
Isomorphic Labs (IsoLabs) was launched in 2021 to advance human health by building on and beyond the Nobel-winning AlphaFold system. Since then, our interdisciplinary team of drug discovery experts and machine learning specialists has built powerful new predictive and generative AI models that accelerate scientific discovery at digital speed. Our