Senior Data & AI Engineer (Databricks Specialist)
Cognida.aiHyderabad, IndiaOn-Site
Needs a free Superjobs account. You'll finish on Cognida.ai's site.
Skill score
Summary
The Senior Data & AI Engineer designs and governs Databricks Lakehouse pipelines with Unity Catalog, and rapidly builds working data and AI demos and POCs for stakeholders and clients. It suits a senior engineer with 10+ years of experience who can work independently across global time zones.
- Function
- Data Science
- Estimated salary
- Est. ₹22–33 LPA
- Estimated for a senior role, ~10 yrs experience, India. Not published by the employer.
- Work mode and location
- On-Site, Hyderabad, India
- Experience
- 10+ yrs
Job description
Overview
A highcaliber Data & AI Engineer who can operate independently while seamlessly collaborating across global time zones.
Position Overview
We are seeking a dynamic, self-driven Data & AI Engineer with deep expertise in the Databricks ecosystem to design, build, and accelerate our next-generation data architecture.
In this role, you won't just write backend pipelines; you will be the bridge between complex data engineering and tangible business value.
A critical measure of success in this role is the ability to rapidly build and execute concrete, high-impact technical demonstrations that showcase the power of unified data and AI.
Because we operate globally, you must thrive in an autonomous, highly flexible environment, collaborating effectively across diverse time zones.
Roles and responsibilities
Databricks Architecture & Governance
Design and implement robust data pipelines utilizing the Databricks Lakehouse platform. Take ownership of data governance, security, and fine-grained lineage using Unity Catalog.
Rapid Prototyping & Demos
Quickly conceptualize, build, and present functional, end-to-end data and AI demos (e.g., live dashboards, real-time pipelines, or LLM/RAG prototypes) to stakeholders and clients.
Core Data Engineering
Build scalable, efficient ETL/ELT pipelines using Spark (PySpark/SQL), Delta Lake, and structured streaming. Ensure optimal performance tuning, partitioning, and cost-management of Databricks clusters.
Global Collaboration & Autonomy
Work independently with minimal supervision, managing your own schedule to align with global, cross-functional teams.
Skills required
Top skills
- Databricks Lakehouse
- Unity Catalog
- Delta Lake
- PySpark
All skills (8)
- SQL
- Python
- Structured Streaming
- Dimensional modeling
- MLflow
- RAG and LLM integration
- Databricks Workflows
- Cloud platforms (AWS/Azure/GCP)
Qualifications
Education : Bachelor's degree , BE/B. Tech in CS or equivalent
Experience : 10+ years
Databricks & Governance Mastery
Proven track record implementing Databricks Unity Catalog for data governance, access control, and data lineage.
Deep understanding of Databricks architecture, including Delta Lake, serverless compute, and Workflows.
Core Data Engineering Excellence
Strong foundational knowledge of data engineering principles (star schemas, dimensional modeling, data warehousing vs. data lakes).
Advanced proficiency in Python/PySpark and SQL.
Experience building robust orchestrations and handling both batch and real-time streaming data.
Experience in building AI/ML workloads (e.g., MLflow, LLM integration, or RAG frameworks) within Databricks.
Agility & Communication
Exceptional ability to build quick, concrete POCs (Proof of Concepts) and technical demos that clearly translate data capabilities into business solutions.
Excellent communication skills, with the ability to clearly articulate technical concepts to both technical and non-technical audiences.
Nice to have
Certifications such as Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
Experience integrating Databricks with enterprise ecosystems, cloud provider native services (AWS/Azure/GCP), or modern BI tools.
Other details
Location : Hyderabad,India
Position : Senior Data & AI Engineer (Databricks Specialist)
Company info
About Cognida.ai
Cognida.ai provides AI and analytics solutions that help businesses make better decisions and run AI in production across data, applications, workflows, integrations and governance, serving enterprises and AI builders. Its Automate platform applies AI to accounting and operations work such as revenue accounting, contract review, invoice processing and PO matching