About me

🔹 About Me: With over 9 years of experience in the IT industry, I am a passionate data engineer specializing in big data batch processing and creating robust data pipelines. My expertise lies in leveraging cutting-edge technologies and frameworks like Spark, AWS, Azure, Airflow, Python, Scala, and the Hadoop ecosystem to deliver scalable and efficient data solutions. 🔹 Technical Skills: I am proficient in a wide range of tools and technologies, including Spark, AWS (S3, EMR, Lambda, CloudFormation, CloudWatch, Redshift, Athena, EC2, AWS Glue), Azure Databricks, Azure Data Factory, and ADLS Gen2. My strong command over SQL enables me to handle complex data manipulations and extract valuable insights. 🔹 Project Highlights: In my first project, I successfully processed search engine impression data, dealing with a massive log volume of 5 to 6 GB per hour. Utilizing low-level Spark API like RDD, I implemented intricate filtering algorithms based on specific business logic. By creating sessional campaigns for targeted products in different regions, I enabled powerful insights, which were then pushed to BI tools for further analysis. This project was executed on-premise, showcasing my ability to handle demanding data processing challenges. In my second project, I spearheaded the processing of credit bureau data for a prominent US-based client. My role involved implementing diverse pipeline strategies to gain comprehensive insights into the US population, including understanding the efficacy of various offers and tracking credit history. This project was hosted on AWS EMR, and I played a significant role in designing the pipeline architecture, collaborating with stakeholders, and training team members. 🔹 How I Add Value: I have made significant contributions in designing and implementing robust data pipelines throughout my career. Working closely with cross-functional teams, I have not only successfully solved complex data-related challenges but have also trained and mentored team members to enhance their skills. If you are looking for a data expert to tackle your data-related problems and architect powerful solutions, let's connect!