Skip to content

Senior Data Engineer

Sumo LogicNoida, Uttar Pradesh, IndiaOn-Site

Estimated salary, not published by the employer: Est. ₹14–21 LPA

Needs a free Superjobs account. You'll finish on Sumo Logic's Greenhouse page.

Skill score

Summary

The Senior Data Engineer designs, builds, and maintains scalable data pipelines and enterprise integrations on a Databricks-based platform, including ingestion frameworks, medallion-layer datasets, Spark and Delta Lake optimization, and Unity Catalog governance. The role suits an experienced engineer who works hands-on with Databricks, PySpark, and AWS and can partner with analytics and business stakeholders.

Function
Data Science
Estimated salary
Est. ₹14–21 LPA
Estimated for a senior role, ~5 yrs experience, India. Not published by the employer.
Work mode and location
On-Site, Noida, Uttar Pradesh, India
Experience
5+ yrs

Job description

Overview

We are looking for a highly skilled Senior Data Engineer to design, build, and maintain scalable data pipelines and integrations within our Databricks-based data platform. The role will focus on expanding our data ecosystem by onboarding new data sources, integrating enterprise applications, and delivering reliable, secure, and analytics-ready datasets.

The ideal candidate will have strong hands-on expertise in Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake, along with experience building production-grade data pipelines and enterprise integrations. Working knowledge of AWS infrastructure and cloud services is important for collaborating with Engineering and other platform teams on connectivity, deployments, and evolving infrastructure requirements.

The candidate will report directly to the Senior Manager - Business Insights & Analytics and collaborate with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders to deliver scalable and reliable data solutions.

Roles and responsibilities

  • Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.
  • Design and implement a medallion architecture (Bronze, Silver, and Gold layers) to deliver trusted, well-structured, analytics-ready datasets.
  • Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems
  • Developing reusable ingestion frameworks to accelerate onboarding of new data sources.
  • Build reliable batch and near-real-time ingestion pipelines with incremental loads, CDC, schema evolution and error recovery.
  • Configure and manage Databricks Workflows and implement ingestion and transformation pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines
  • Optimize Apache Spark and Delta Lake workloads using execution-plan analysis, efficient join strategies, data-skew handling, partitioning, and SQL query tuning to improve performance and control compute costs.
  • Implement Unity Catalog governance, including access controls, lineage, auditing, and sensitive-data protection through masking and row-level security.
  • Maintain and support existing data workflows running on Amazon Redshift and Amazon EC2, collaborating with Engineering and IT/Cloud teams on connectivity, IAM permissions, S3 access, deployments, and operational issues.
  • Follow engineering best practices, including version control, code reviews, automated testing, CI/CD, and technical documentation.
  • Partner with Data Analytics and business stakeholders to translate requirements into trusted, well-documented, analytics-ready datasets.
  • Provide technical guidance to team members and identify opportunities to improve automation, engineering standards, platform reliability, and resource utilization.

Skills required

Top skills

  • Databricks
  • Apache Spark
  • PySpark
  • Python

All skills (8)

  • SQL
  • Delta Lake
  • AWS services
  • REST APIs
  • Unity Catalog
  • Auto Loader
  • Lakeflow Spark Declarative Pipelines
  • Amazon Bedrock

Qualifications

  • 5+ years of experience in data engineering or a related technical role, including developing and supporting production data pipelines.
  • Strong hands-on expertise in Databricks, including developing, troubleshooting, and optimizing production workloads.
  • Strong proficiency in Python/PySpark, SQL, and Apache Spark for scalable data processing and transformation.
  • Practical experience with Delta Lake, including table management, incremental processing, and performance optimization.
  • Experience with data architecture and modeling, including medallion architecture and the development of analytics-ready datasets.
  • Experience with Databricks Workflows or equivalent orchestration solutions.
  • Proven experience building data ingestion pipelines and integrating REST APIs, databases, and enterprise applications.
  • Understanding of reliable data processing patterns, including retries, idempotency, schema evolution, backfills, and failure recovery.
  • Working knowledge of AWS services and cloud infrastructure, particularly IAM roles and policies, S3 storage and access management, and secure connectivity between source systems and data platforms.
  • Experience with Git/GitHub, code reviews, automated testing, and CI/CD practices.
  • Strong problem-solving skills and the ability to independently design, build, deploy, and maintain production data solutions.
  • Good communication and collaboration skills to work effectively with Data Analytics, Engineering, IT/Cloud, Security, and business stakeholders.

Nice to have

  • Experience with Databricks Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables), Auto Loader, and advanced incremental ingestion patterns.
  • Experience implementing Unity Catalog governance, including fine-grained access controls, data lineage, auditing, and sensitive-data protection.
  • Experience with Databricks Genie, Genie Agents, and Genie Code
  • Experience in AI/ML, Data Science, LLMs, contextualization, Amazon Bedrock

Company info

Sumo Logic is a SaaS log analytics platform, now sold as its Intelligent Operations Platform, that unifies security and operational data so teams can detect, investigate, and resolve cybersecurity and cloud operations issues. Its customers are digital teams and organizations that use it to protect against security threats and keep

More jobs at Sumo Logic