•Designed and implemented scalable data pipelines: Developed and optimized ETL processes using Apache Spark and Teradata SQL to handle large volumes of structured data •Data warehousing: Built and maintained data warehouses on platforms such as Data Lake, Teradata ensuring high performance and reliability for business intelligence needs. •Programming and scripting: Utilized Python, SQL, and Shell Scripting for data processing, analysis, and automation tasks. •Cloud data infrastructure: Migrated on-premises data systems to cloud platforms, leveraging cloud-native services for improved scalability and cost-efficiency. •Monitoring and troubleshooting: Developed monitoring and alerting solutions for data pipelines and infrastructure, ensuring high availability and quick issue resolution. •Documentation and training: Created comprehensive documentation for data workflows and provided training sessions for team members on new technologies and best practices. •Data transformation: Applied complex transformation logic, including data cleaning, CDC and enrichment, using Python and SQL to ensure data quality and prepare it for analysis. •Loading and optimization: Implemented efficient loading strategies into data warehouses (Teradata) and data lakes (AWS S3), optimizing for speed and reducing storage costs. •ETL automation: Automated ETL workflows using Apache Airflow, scheduling and orchestrating data processing tasks to run reliably and reduce manual effort.