Data Engineer (SDE 2)
FamPayBengaluruOn-Site
Needs a free Superjobs account. You'll finish on FamPay's Lever page.
Skill score
Summary
The Data Engineer II will build and scale a high-performance lakehouse with CDC ingestion, distributed compute, and orchestration, while designing OLAP domain models and partnering with product and business stakeholders. The role suits an engineer with 3 to 5 years in distributed data systems who is hands-on with PySpark, SQL, AWS, and AI-assisted development.
- Function
- Data Science
- Estimated salary
- Est. ₹9.5–14 LPA
- Estimated for a mid-level role, ~3 yrs experience, India. Not published by the employer.
- Work mode and location
- On-Site, Bengaluru
- Experience
- 3–5 yrs
- Employment type
- Full-time
Job description
Overview
About the role
We are looking for a Data Engineer II (SDE-2) to join our data team. The ideal candidate will be a play a key role to develop of high performant and scalable Data Lake-house, moving us toward a world of sub-minute data latency and unified batch/streaming compute. This is an engineering-heavy role where you will manage complex CDC flows, optimize distributed query engines and leverage AI to accelerate our development lifecycle.
Skills required
Top skills
- Python
- PySpark
- SQL
- AWS
All skills (8)
- Infrastructure as Code
- Airflow
- Temporal
- Dimensional modelling
- Go
- Java
- Scala
- Debezium
Qualifications
- Experience: 3–5 years in Data Engineering, specifically with distributed systems and cloud-native architectures.
- Coding: Expert-level Python/PySpark and SQL. Familiarity with Go/Java/Scala is a plus
- Infrastructure: Hands-on experience with AWS (S3, EKS, MSK) and Infrastructure-as-Code.
- Orchestration: Experience with Airflow or Temporal for complex workflow management.
- AI-Native: Proficiency in using AI tools (Claude, Codex, Copilot) to write, test, and document code efficiently.
- Systems Thinking: Ability to explain the trade-offs between different storage formats and processing frameworks.
- Tech Execution : Drive key tech initiatives by preparing TRD and actively involve in design reviews.
- Domain Modelling - Should be hands on in designing Domain models for OLAP like Fact, Dimension, Cumulative , types of SCD’s and OBT pattern tables.
- Self Starter - Lead the team technically and bring in new ideas to contribute to the growth of the charter.
- Stakeholder Interaction - Interact with the Product & Key Stakeholders & help them by adding value to the business workflow with data & analytics.
Nice to have
- Real-time CDC: Ownership of high-throughput ingestion from RDBMS to Lakehouse using Debezium, PeerDB.
- Lakehouse Architecture: Designing and optimizing table formats (Iceberg, Delta, Hudi) for both performance and storage efficiency.
- Unified Compute: Developing robust ETL/ELT frameworks in PySpark and Flink (handling both batch and streaming workloads).
- Infrastructure & Ops: Managing data workloads on AWS (EMR, EKS, MSK, S3) and automating everything via Gitlab/Github Actions.
- Query & BI: Tuning Trino or Clickhouse to power real-time dashboards in Metabase, Superset, and PowerBI.
Other details
- Ingestion & CDC: OLake and PeerDB for near real-time sync from production systems into Alchemy; Debezium/Kafka for CDC-heavy use cases, with support for Kafka and S3-based sources.
- Lakehouse / Storage: Alchemy on Apache Iceberg with S3 as the data lake storage layer and AWS Glue Catalog for metadata; exposure to Delta/Hudi is a plus.
- Processing & Compute: PySpark on EMR/EKS for batch and streaming workloads; Flink and Spark Structured Streaming fundamentals for low-latency pipelines.
- Streaming Platform: MSK / Kafka for event-driven ingestion, CDC propagation, replay, backfills, and operational monitoring through Kafka UI.
- Query & Serving Layer: Trino over Alchemy/Iceberg for lakehouse analytics, ClickHouse for high-throughput operational and real-time dashboards, and BigQuery exposure where applicable.
- Workflow Orchestration: Airflow for scheduled data pipelines, DQ/reconciliation DAGs, backfills, and SLA-driven jobs; Temporal for durable workflow execution in ingestion services.
- Data Quality & Governance: DQ checks, freshness/SLA monitoring, source-to-lake reconciliation, deduplication, schema evolution handling, and cataloging/lineage through OpenMetadata.
- Infrastructure & DevOps: AWS (S3, EKS, EMR, MSK), Kubernetes, Terraform/IaC, GitLab/GitHub Actions, observability via Grafana/CloudWatch, and production runbook discipline.
- BI & Analytics: Metabase, Superset, Tableau, and PowerBI for business dashboards; strong ability to model curated marts, fact/dimension tables, SCDs, and OBT patterns for stakeholder-facing analytics.
Benefits
- Relocation assistance to make your move seamless.
- Free office meals (lunch & dinner).
- Generous leave policy, including birthday leave, period leave, paternity and maternity support, and more.
- Salary advance and loan policies for any financial help.
- Quarterly rewards and recognition programs, and a referral program with great incentives.
- Access the latest gadgets and tools.
- Comprehensive health insurance for you and your family, mental health support.
- Tax benefits with options like food coupons, phone allowances, car/device leasing.
- Retirement perks like PF contribution, leave encashment and gratuity.
PS - We’re looking for someone with an AI-native mindset, who actively leverages AI in their day-to-day work, as we encourage and embed it across every level at Fam.
Company info
About FamPay
Fam is a payments app in India that lets users make online and offline payments through UPI and its FamCard, with a flagship Spending Account, FamX, for managing and saving money. It is used by people above 11 years old in India who want to make UPI and card payments.