I am Soumil Nitin Shah, a highly skilled Lead Data Engineer specializing in AWS, PySpark, and building scalable data platforms. My expertise lies in developing innovative solutions like the "LakeBoost" framework, which has significantly enhanced data lake efficiency by integrating Hudi and Glue ETL, reducing operational overhead by 40%. With a strong commitment to knowledge sharing, my YouTube channel has garnered 42,000 subscribers, where I create instructional videos on big data, AWS, and various data engineering topics. My career is marked by a passion for crafting robust data ingestion frameworks, managing over 2TB of data monthly, and orchestrating thousands of jobs. At JobTarget, I transitioned our data processing approach to Hudi's incremental ETL, drastically improving efficiency and reducing costs by 4-5x. My experience also includes developing the "Sourcer" web application, leveraging Elasticsearch to optimize search performance, resulting in an 80x improvement. I hold a Master's in Computer Engineering and Electrical Engineering from the University of Bridgeport and a Bachelor's in Electronic Engineering from the University of Mumbai. I am proficient in Spark SQL, Python, Apache X Table, EMR, DynamoDB, AWS Glue, and more, always striving to push the boundaries of data engineering.