Skip to content

Big Data Engineer- Scala

CAPCOIndia - PuneOn-Site

Estimated salary, not published by the employer: Est. ₹9.5–14 LPA

Needs a free Superjobs account. You'll finish on CAPCO's Greenhouse page.

Skill score

Summary

The Big Data Engineer designs, builds and tunes scalable data-processing pipelines in Scala and Apache Spark, working with Hadoop, Hive and Kafka on large structured and unstructured datasets. It suits an experienced engineer who is hands-on with distributed data platforms and software engineering practices.

Function
Data Science
Estimated salary
Est. ₹9.5–14 LPA
Estimated for a mid-level role, ~3 yrs experience (typical for the level), India. Not published by the employer.
Work mode and location
On-Site, India - Pune

Job description

Overview

Job Description: Big Data Engineer – Scala

We are looking for an experienced Big Data Engineer with strong expertise in Scala and Apache Spark to design, develop, and optimize scalable data-processing solutions. The ideal candidate should have hands-on experience with distributed data platforms, large-scale ETL/data pipelines, and big data technologies.

Roles and responsibilities

  • Design, develop, and maintain scalable Big Data applications and data pipelines.
  • Develop high-performance data-processing solutions using Scala and Apache Spark.
  • Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data.
  • Work with technologies such as Hadoop, HDFS, Hive, Kafka, and Spark SQL.
  • Perform Spark job performance tuning and optimization.
  • Develop reusable and maintainable data-processing frameworks.
  • Troubleshoot production issues related to data pipelines and distributed applications.
  • Implement data-quality checks, monitoring, and error-handling mechanisms.
  • Collaborate with data architects, analysts, business teams, and other engineering teams.
  • Follow software engineering best practices, including Git, code reviews, testing, and CI/CD.

Skills required

Top skills

  • Scala
  • Apache Spark
  • Spark SQL
  • SQL

All skills (8)

  • Hadoop
  • HDFS
  • Hive
  • Kafka
  • ETL pipelines
  • Parquet, Avro, JSON
  • Databricks
  • Airflow

Qualifications

  • Strong hands-on programming experience in Scala.
  • Strong experience with Apache Spark / Spark SQL.
  • Good understanding of Big Data and distributed computing concepts.
  • Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
  • Experience building large-scale ETL/data engineering pipelines.
  • Strong SQL skills.
  • Experience working with Kafka or similar messaging/streaming technologies.
  • Understanding of data formats such as Parquet, Avro, and JSON.
  • Experience with performance optimization and troubleshooting of Spark applications.
  • Good understanding of software development lifecycle and version-control systems.

Nice to have

  • Experience with AWS, Azure, or GCP.
  • Knowledge of Databricks.
  • Experience with Airflow or similar orchestration tools.
  • Exposure to Delta Lake or other modern lakehouse technologies.
  • Knowledge of Docker, Kubernetes, and CI/CD pipelines.
  • Experience working in Agile/Scrum environments.

Other details

Scala | Apache Spark | Spark SQL | SQL | Hadoop | HDFS | Hive | Kafka | ETL | Data Engineering | Big Data

Company info

“Capco, a Wipro company, is a global technology and management consulting firm. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors.

More jobs at CAPCO