Senior Data Engineer | Remote | 5–8 Years | Working Hours: Monday to Friday, 2:00 PM – 11:00 PM IST

Senior Data Engineer

Location: Remote

Experience: 5–8 Years

Working Hours: Monday to Friday, 2:00 PM – 11:00 PM IST

UAN: Mandatory

About the Role

We are looking for a highly skilled and experienced Senior Data Engineer to join our growing data team. The ideal candidate will have strong expertise in Snowflake, AWS, Python, SQL, DBT, and Luigi, with proven experience in building scalable data platforms, developing robust ETL/ELT pipelines, and leading cloud migration initiatives.

This role requires a hands-on technical leader who can architect modern data solutions, optimize performance, mentor team members, and collaborate effectively with cross-functional stakeholders.

Key Responsibilities

Snowflake Migration & Data Platform Modernization

• Lead end-to-end Snowflake migration projects, including assessment, planning, implementation, testing, and optimization.
• Design scalable Snowflake data architectures and data models.
• Ensure seamless migration with complete data integrity and minimal business disruption.
• Optimize Snowflake performance through clustering, partitioning, and query tuning.
• Implement Snowflake best practices for security, governance, and scalability.

Data Pipeline Development & Orchestration

• Design, build, and maintain scalable ETL/ELT pipelines using Python and DBT.
• Develop and orchestrate complex workflows using Luigi.
• Process and transform large-scale datasets, including clickstream, demographic, and business data.
• Implement data validation, quality checks, and reliability frameworks.

AWS Cloud Infrastructure Management

• Configure, deploy, and manage AWS services including EC2, S3, RDS, and EMR.
• Develop secure, scalable, and cost-efficient cloud-based data solutions.
• Support data ingestion, storage, transformation, and retrieval processes.
• Monitor and optimize AWS infrastructure performance and availability.

Legacy System Modernization

• Enhance and maintain existing data platforms and frameworks.
• Identify opportunities for modernization and cloud migration.
• Collaborate with business and technical stakeholders to align solutions with organizational goals.

Monitoring & Performance Optimization

• Build monitoring frameworks, dashboards, and alerting mechanisms.
• Troubleshoot data pipeline failures, infrastructure issues, and performance bottlenecks.
• Conduct root-cause analysis and implement preventive measures.
• Ensure high availability and reliability of data systems.

Technical Leadership & Documentation

• Create and maintain technical documentation, workflows, and operational runbooks.
• Establish engineering standards and best practices.
• Mentor junior engineers and provide technical guidance.
• Drive knowledge-sharing initiatives across teams.

Required Qualifications

Experience

• 5–8 years of experience in Data Engineering, Data Platform Engineering, or related roles.
• Proven experience leading large-scale data engineering and migration projects.

Snowflake Expertise

• Minimum 3 years of hands-on Snowflake experience.
• Strong knowledge of:
• Database design and schema architecture
• Snowpipe and COPY INTO data loading techniques
• Query optimization and performance tuning
• Security, roles, permissions, and access control
• Data sharing and Snowflake best practices

Technical Skills

• Strong programming expertise in Python.
• Advanced SQL development and query optimization skills.
• Hands-on experience with DBT for data transformation and modeling.
• Experience with Luigi workflow orchestration.
• Linux shell scripting and automation experience.

Data Engineering & Big Data

• Strong understanding of ETL/ELT concepts and implementation.
• Experience with Hadoop ecosystem technologies.
• Knowledge of large-scale data processing frameworks.
• Strong understanding of data warehousing concepts and best practices.

AWS Cloud Services

Hands-on experience with:

• Amazon EC2
• Amazon S3
• Amazon RDS
• Amazon EMR

Monitoring & Operations

• Experience implementing monitoring, logging, and alerting solutions.
• Strong troubleshooting and production support capabilities.

Soft Skills

• Excellent communication and stakeholder management skills.
• Strong leadership and mentoring abilities.
• Ability to work independently in a remote environment.
• Strong analytical and problem-solving mindset.

Preferred Qualifications

• Experience with Snowflake migration tools such as Fivetran, Matillion, or similar platforms.
• Knowledge of Snowflake governance and security features:
• Dynamic Data Masking
• Row Access Policies
• External Tokenization
• Experience implementing CI/CD pipelines for data engineering workflows.
• Familiarity with automated testing and deployment of Snowflake objects using DBT.
• Knowledge of data governance, security, and compliance frameworks.
• Experience with Spark, Hive, or other big data technologies.
• Exposure to analytics, data science workflows, DevOps practices, and infrastructure automation.

Success Factors

• Strong ownership and accountability.
• Ability to independently lead technical initiatives.
• Passion for building scalable, reliable, and high-performance data platforms.
• Commitment to engineering excellence and continuous improvement.
• Strong collaboration skills with technical and business stakeholders.