Big Data Engineer- Scala
CAPCOIndia - PuneOn-Site
Needs a free Superjobs account. You'll finish on CAPCO's Greenhouse page.
Skill score
Summary
The Big Data Engineer designs, builds and tunes scalable data-processing pipelines in Scala and Apache Spark, working with Hadoop, Hive and Kafka on large structured and unstructured datasets. It suits an experienced engineer who is hands-on with distributed data platforms and software engineering practices.
- Function
- Data Science
- Estimated salary
- Est. ₹9.5–14 LPA
- Estimated for a mid-level role, ~3 yrs experience (typical for the level), India. Not published by the employer.
- Work mode and location
- On-Site, India - Pune
Job description
Overview
Job Description: Big Data Engineer – Scala
We are looking for an experienced Big Data Engineer with strong expertise in Scala and Apache Spark to design, develop, and optimize scalable data-processing solutions. The ideal candidate should have hands-on experience with distributed data platforms, large-scale ETL/data pipelines, and big data technologies.
Roles and responsibilities
- Design, develop, and maintain scalable Big Data applications and data pipelines.
- Develop high-performance data-processing solutions using Scala and Apache Spark.
- Build and optimize ETL/ELT pipelines for large volumes of structured and unstructured data.
- Work with technologies such as Hadoop, HDFS, Hive, Kafka, and Spark SQL.
- Perform Spark job performance tuning and optimization.
- Develop reusable and maintainable data-processing frameworks.
- Troubleshoot production issues related to data pipelines and distributed applications.
- Implement data-quality checks, monitoring, and error-handling mechanisms.
- Collaborate with data architects, analysts, business teams, and other engineering teams.
- Follow software engineering best practices, including Git, code reviews, testing, and CI/CD.
Skills required
Top skills
- Scala
- Apache Spark
- Spark SQL
- SQL
All skills (8)
- Hadoop
- HDFS
- Hive
- Kafka
- ETL pipelines
- Parquet, Avro, JSON
- Databricks
- Airflow
Qualifications
- Strong hands-on programming experience in Scala.
- Strong experience with Apache Spark / Spark SQL.
- Good understanding of Big Data and distributed computing concepts.
- Experience with Hadoop ecosystem technologies, including Hadoop/HDFS and Hive.
- Experience building large-scale ETL/data engineering pipelines.
- Strong SQL skills.
- Experience working with Kafka or similar messaging/streaming technologies.
- Understanding of data formats such as Parquet, Avro, and JSON.
- Experience with performance optimization and troubleshooting of Spark applications.
- Good understanding of software development lifecycle and version-control systems.
Nice to have
- Experience with AWS, Azure, or GCP.
- Knowledge of Databricks.
- Experience with Airflow or similar orchestration tools.
- Exposure to Delta Lake or other modern lakehouse technologies.
- Knowledge of Docker, Kubernetes, and CI/CD pipelines.
- Experience working in Agile/Scrum environments.
Other details
Scala | Apache Spark | Spark SQL | SQL | Hadoop | HDFS | Hive | Kafka | ETL | Data Engineering | Big Data
Company info
About CAPCO
“Capco, a Wipro company, is a global technology and management consulting firm. With our presence across 32 cities across globe, we support 100+ clients across banking, financial and Energy sectors.