Data Engineering Project

Dive into this comprehensive data engineering project that extracts stock market data from Yahoo Finance, processes it using AWS Batch and PySpark, and performs transformations with AWS Glue.

Key Features:

• Data Ingestion from Yahoo Finance API
• Batch Processing with AWS Batch and Python
• Transformations using PySpark in AWS Glue
• Implementation of SCD Type 2
• Failure Notification Pipeline
• End-to-End Orchestration for Stock Data Handling

Ideal for aspiring data engineers and professionals seeking real-world project experience.

GitHub Repository: https://github.com/Raiyankarim/Yahoo_Finance_DataEngineeering_Project