Staff Engineer - Kafka Codebase and Internals
Phone PeBengaluru, Karnataka, IndiaOn-Site
Needs a free Superjobs account. You'll finish on Phone Pe's SmartRecruiters page.
Skill score
Summary
This Staff Engineer works directly on the Apache Kafka codebase to maintain PhonePe's internal Kafka distribution, contributing upstream, backporting fixes, and advising platform teams on broker internals. It suits a senior engineer with deep Kafka internals knowledge, upstream contribution history, and systems-level expertise who wants technical ownership rather than operations work.
- Function
- Development
- Estimated salary
- Est. ₹18–28 LPA
- Estimated for a staff-level role, ~8 yrs experience, India. Not published by the employer.
- Work mode and location
- On-Site, Bengaluru, Karnataka, India
- Experience
- 8+ yrs
- Employment type
- Full-time
Job description
Overview
About the Role
We are building an in-house team to maintain and contribute to our internal Apache Kafka distribution. This is not an operations role — you will work on the Kafka codebase itself: the broker, the KRaft controller, the protocol, the clients, and the storage layer. You will be our technical anchor for all things Kafka internals, driving upstream contributions, backporting critical fixes, and shaping our internal distribution to meet the needs of a large-scale production fleet.
We already do this successfully with GlusterFS, where we maintain an in-house distribution backed by deep upstream expertise. We want to replicate that model for Kafka.
Roles and responsibilities
- Own our internal Kafka distribution: branch management, patch curation, release cutting, upgrade planning
- Contribute upstream to Apache Kafka — bug fixes, KIPs, code reviews, design discussions
- Diagnose and fix deep broker/controller issues: replication, partition reassignment, KRaft consensus, log storage, ISR management, protocol handling
- Build custom patches for production needs that aren't suitable for upstream (performance tuning for our workload profile, observability hooks, operational safety valves)
- Establish engineering practices for Kafka development within the team: how we test, how we benchmark, how we validate patches before rolling them out
- Mentor a supporting engineer and grow the team's capability over time
- Act as the internal authority on Kafka internals — advise platform teams on configuration, capacity planning, and architectural decisions that require knowledge of broker behavior
- Act as the primary interface for audit and compliance teams, ensuring PhonePe Kafka distribution meets various compliance and regulatory standards
Skills required
Top skills
- Apache Kafka internals
- Distributed systems
- KRaft consensus
- JVM internals
All skills (8)
- Linux I/O and networking
- Upstream open source contribution
- Fork maintenance
- Kafka committer or PMC
- Kafka Streams internals
- Kafka Connect internals
- KIP authoring
- Kafka test infrastructure
Qualifications
What are we looking for?
- Deep familiarity with the Kafka codebase — you have read, modified, and debugged the broker, controller, or client internals. Not just operated Kafka, but worked on it.
- Track record of upstream contribution — committed patches, participated in KIP discussions, reviewed PRs, or equivalent engagement with the Apache Kafka community. Active committer/PMC status is strongly preferred but not strictly required if your contribution history demonstrates equivalent depth.
- Strong distributed systems fundamentals — consensus protocols, replication, exactly-once semantics, log-structured storage, distributed state machines. You understand these not as buzzwords but as engineering tradeoffs you've navigated.
- Systems-level programming comfort — JVM internals (GC tuning, memory layout, JIT behavior), OS-level I/O (page cache, fsync semantics, zero-copy), networking (TCP tuning, socket buffers). Kafka performance work lives at this layer.
- Experience maintaining a fork or distribution — you've managed the tension between carrying local patches and staying mergeable with upstream. You know what it costs to diverge and how to minimize that cost. •
- 8+ years of experience in systems/infrastructure software engineering, with a meaningful portion spent on Kafka or comparable distributed systems (Pulsar, Bookkeeper, Zookeeper, etcd, CockroachDB, etc.)
Nice to have
Good to have
- Apache Kafka Committer or PMC member
- Experience with KRaft (the ZooKeeper-free controller)
- Authored or co-authored a KIP
- Experience with Kafka Streams or Kafka Connect internals (not just usage)`
- Familiarity with Kafka's test infrastructure (system tests, Trogdor, ducktape)
Company info
About Phone Pe
PhonePe Limited (Formerly PhonePe Private Limited) is a technology company that builds digital platforms for Payments, Digital Distribution Services and Financial Services. Headquartered in India, the PhonePe digital payments app was launched in 2016. As of April 2026, PhonePe has over 70 Crore life-till-date registered users and a digital payments