Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
MastercardPune, IndiaOn-Site
Needs a free Superjobs account. You'll finish on Mastercard's site.
Skill score
Summary
The Principal Data Engineer designs and builds scalable, governed data platforms, pipelines, and reusable data products that support analytics, machine learning, generative AI, and agentic AI in an R&D setting. The role suits a senior, hands-on technical leader who can turn ambiguous research goals into production-ready systems and mentor other engineers.
- Function
- Data Science
- Estimated salary
- Est. ₹28–43 LPA
- Estimated for a staff-level role, ~12 yrs experience, India. Not published by the employer.
- Work mode and location
- On-Site, Pune, India
- Experience
- 12–18 yrs
- Employment type
- Full-time
Job description
Overview
Principal Data Engineer (AWS, Databricks, Ai, ML Flow, Data Architecture, Apache Airflow)
Overview
Mastercard Foundry R&D develops emerging technology solutions and transforms promising concepts into scalable products and platforms.
We are seeking a Principal Data Engineer to design and build the data foundations that power advanced analytics, machine learning, generative AI, and agentic AI solutions. This role requires a hands-on technical leader who can turn ambiguous R&D objectives into secure, scalable, and production-ready data systems.
You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to create reusable data capabilities that accelerate experimentation and the delivery of innovative products.
Roles and responsibilities
What You'll Do
- Lead the architecture, design, and engineering of scalable batch, streaming, and event-driven data platforms.
- Design and build reliable data ingestion, transformation, enrichment, processing, and consumption pipelines.
- Develop governed lakehouse and modern data platform architectures using cloud-native technologies and distributed data processing frameworks.
- Build and operate platforms that support analytics, machine learning, generative AI, and agentic AI workloads.
- Create reusable data products, services, APIs, and self-service capabilities that accelerate experimentation and production delivery.
- Enable the full AI and machine learning lifecycle through robust feature engineering, training, evaluation, deployment, and inference pipelines.
- Optimize data workloads for performance, scalability, reliability, resiliency, and cost efficiency.
- Define engineering standards for data modeling, software development, testing, CI/CD, observability, and operational excellence.
- Implement data governance capabilities including quality controls, lineage, metadata management, cataloging, access control, retention, and compliance.
- Apply security and privacy-by-design principles for sensitive and regulated data environments.
- Evaluate emerging data and AI technologies through prototypes, proof-of-concepts, and technical assessments.
- Translate loosely defined research, innovation, or product requirements into practical architectures and incremental delivery plans.
- Provide hands-on technical leadership through architecture reviews, code reviews, troubleshooting, and engineering mentorship.
- Partner with cross-functional teams to mature successful R&D initiatives into enterprise-grade production solutions.
- Communicate technical decisions, trade-offs, risks, dependencies, and roadmap recommendations to both technical and business stakeholders.
- Drive adoption of modern data engineering practices, platform automation, and platform reliability disciplines.
Skills required
Top skills
- Python
- PySpark
- Advanced SQL
- Apache Spark
All skills (10)
- Lakehouse platforms
- AWS
- Apache Airflow
- CI/CD
- Infrastructure as Code
- Data governance
- Azure
- MLflow
- Databricks Workflows
- Kafka
Qualifications
Required Qualifications
- Typically 12-18 years of overall career relevant experience, including significant ownership of complex enterprise data engineering solutions.
- Extensive experience designing, building, and operating large-scale production data platforms.
- Expert programming skills in Python, PySpark, and modern software engineering practices.
- Advanced SQL expertise, including data modeling, performance tuning, query optimization, and large-scale analytical processing.
- Strong hands-on experience with distributed data processing technologies such as Apache Spark and modern lakehouse platforms.
- Experience building and operating cloud-native data platforms in AWS, or Azure, or other enterprise cloud environments.
- Strong knowledge of modern data architecture patterns, including data lakes, lakehouses, data meshes, data products, and event-driven architectures.
- Experience with cloud storage, data integration, streaming, serverless, observability, and security services across public cloud platforms.
- Experience building scalable batch and real-time data pipelines.
- Experience with workflow orchestration platforms such as Apache Airflow, Databricks Workflows, AWS Step Functions, Azure Data Factory, or similar technologies.
- Experience implementing CI/CD, automated testing, version control, Infrastructure as Code, and platform automation.
- Strong understanding of data governance, metadata management, lineage, quality frameworks, privacy controls, and access management.
- Experience diagnosing and resolving complex performance, reliability, scalability, and operational challenges.
- Ability to make sound architectural decisions while balancing delivery speed, innovation, maintainability, security, and cost.
- Proven ability to thrive in R&D and innovation-focused environments where priorities and requirements may evolve through experimentation.
- Strong communication, collaboration, technical leadership, and mentoring skills.
- Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline, or equivalent practical experience.
Nice to have
- Experience supporting machine learning, generative AI, or agentic AI systems.
- Experience with MLflow or comparable ML lifecycle tooling.
- Experience with Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
- Experience with Unity Catalog, Microsoft Purview, or comparable catalog and governance platforms.
- Experience with Kafka or other event-streaming technologies.
- Experience with containerized and cloud-native platforms, including Docker and Kubernetes.
- Experience with Terraform or another Infrastructure as Code framework.
- Experience integrating structured, semi-structured, unstructured, and streaming data.
- Knowledge of responsible AI, model evaluation, and AI platform observability.
- Experience in payments, financial services, or another regulated industry.
Other details
Success in This Role
Success requires someone who
- Remains strongly hands-on while providing technical direction.
- Can build production-quality systems without introducing unnecessary platform complexity.
- Converts experimentation into reusable engineering capabilities.
- Designs for security, governance, reliability, and observability from the outset.
- Challenges assumptions and validates architectural choices through evidence and prototypes.
- Enables AI and product teams rather than taking ownership of data science or model research.
- Influences across teams without depending on formal people-management authority.
Company info
About Mastercard
Mastercard is a global payments company that provides payment transaction processing and related payment services, including credit, debit and prepaid card payments that let people, businesses and governments transact securely. Its products and services are used by consumers, merchants and card-issuing banks and credit unions worldwide.
Recent news
Mastercard Expands Crypto Card Program with New Partnerships
Mastercard Appoints Former PayPal Exec as New Chief Financial Officer
Mastercard Unveils New Sustainable Card Program