Skip to content

Principal/Staff Data Engineer

HG InsightsPune, Maharashtra, IndiaOn-Site

Estimated salary, not published by the employer: Est. ₹33–49 LPA

Needs a free Superjobs account. You'll finish on HG Insights's site.

Skill score

Summary

The Principal Data Engineer is a senior individual contributor who owns the architecture of HG's data platform, from raw vendor file ingestion through transformation, entity matching, and delivery of finished datasets. The role suits an experienced staff-level engineer who is strong in Spark and Databricks, entity resolution, and building reliable, cost-aware pipelines.

Function
Data Science
Estimated salary
Est. ₹33–49 LPA
Estimated for a staff-level role, ~15 yrs experience, India. Not published by the employer.
Work mode and location
On-Site, Pune, Maharashtra, India
Experience
15+ yrs

Job description

Overview

This is a senior individual contributor role on the team that builds and runs HG's data platform: everything between a raw vendor file landing in our lake and a finished dataset arriving in a customer's hands. It includes the transformation stack that turns messy external signals into product-grade company attributes, the matching systems that decide which company a piece of evidence belongs to and the internal applications our analysts use to curate and correct the result.

About the team

You will be based in Pune, working with engineers at our Pune, US & Brazil locations.

Roles and responsibilities

  • Own the architecture of the data platform end to end, and make the calls on build vs. buy, batch vs. incremental, and where each workload belongs.
  • Build the quality layer that catches silent failure: contracts at every handoff, freshness and completeness monitoring, and pipelines that quarantine bad data rather than publish it.
  • Improve entity matching and attribution — and make match decisions explainable to the customers who depend on them.
  • Make our release cycle boring. Recurring deliveries should not depend on people watching them.
  • Treat compute cost as a first-class engineering metric across Spark, orchestration, and the serving tier.

Skills required

Top skills

  • Spark
  • Databricks
  • SQL
  • Dimensional modelling

All skills (8)

  • Entity resolution
  • Airflow
  • AWS EMR
  • AWS EC2
  • AWS S3
  • Python
  • MySQL
  • Postgres

Qualifications

  • 15+ years building production grade data engineering systems, with a minimum of 5 years in a staff or principal scope.
  • Deep Spark and Databricks expertise at multi-terabyte scale, including the operational side of lakehouse table formats — merges, compaction, small files, schema evolution.
  • Strong SQL and dimensional modelling, and the judgment to know when to break the rules.
  • Experience with performant and scalable OLTP setups (MySQL/Postgres).
  • Airflow orchestration (DAGs, operators, sensors) and integration with Spark/Databricks.
  • Proven experience in AWS ecosystems (EC2, S3, EMR).
  • Hands-on entity resolution, fuzzy matching, or record linkage at scale. This is central to what we do.
  • Python/Scala/Java, with real software discipline: testing, CI/CD, code review, infrastructure as code is a plus.

Nice to have

  • Experience in Docker - Kubernetes, Terraform is a plus.
  • Experience in integrating AI first implementations in traditional data engineering setups will greatly help in shaping our future designs.
  • Experience with machine learning pipelines (Spark MLlib, Databricks ML) for predictive analytics.
  • Knowledge of data governance frameworks and compliance standards (GDPR, CCPA).

Preferred Qualifications

B2B firmographic, technographic or intent data; streaming and CDC; warehouse-to-lakehouse migration experience.

Other details

We take care of our people fully and genuinely. Every HG Insights employee receives comprehensive healthcare coverage for themselves and their families, a monthly wellness reimbursement and access to Able To professional mental health coaching for stress, anxiety and depression support as per the regions. For your financial future, US employees benefit from a 401(k) with up to 5% company matching and all UK employees are enrolled in a pension scheme. Every employee also receives stock options because the people who build this company should share in its success.

We invest in your growth through an annual learning reimbursement and a genuine commitment to developing your career at every stage. And through our Culture Club, you will always have a real voice in shaping the kind of workplace we build together.

Company info

HG Insights is a contextual intelligence platform that gives B2B go-to-market teams and their AI agents a unified view of markets, accounts, and buyers, combining technology usage, IT spend, competitor, and buyer intent data. Its customers include GTM, marketing, RevOps, and sales teams at technology companies that use the platform

More jobs at HG Insights