As an experienced Hadoop Ecosystem and Apache Spark developer, I have spent over a decade working with big data ecosystems. With proficiency in Scala, Python, and Java-based frontend, backend, and middleware services, I have a proven track record of managing and troubleshooting Cloudera and Mesosphere clusters, handling day-to-day issues with platforms, and implementing data load workloads using Impala, Sqoop, and Apache Kudu as the relational storage. My expertise extends to a wide range of technologies, including Apache Spark, Spark MLLib, Azure Machine Learning, Azure Data Factory, HDInsight, ML Flow, Azure DevOps, Impala, Streamsets, MapReduce, Hive, Pig, Sqoop, and more. I have also demonstrated my ability to set Data Architecture and Machine learning architecture on Azure cloud, and I have advised clients on the Hadoop cluster size, helping them to set up Cloudera and Hortonworks Hadoop stack. I take pride in my ability to guide a team of data engineers to build scalable and robust data pipelines, and I have provided innovative tools for the team to get the data science done in the correct way. I am also passionate about mentoring juniors in the data engineering domain, and I have been involved in requirement engineering with different business units to build appropriate data solutions. Throughout my career, I have implemented various use cases such as: - Fraud Detection for insurance clients - Multiple use cases for the airline industry - Inventory Management, including predicting box sizes, for fashion clients - Predictive Maintenance for industrial clients - Predicting optimal temperature for assets for energy I fairly experienced in Reporting, Retail Banking, Call Center Survey Management, Airline, and Energy, and have developed data quality check solutions to early detect data quality issues and find appropriate solutions to handle them. I have visited Spark-based AI conferences to bring new and innovative tools to the team to resolve existing issues (e.g., ML Flow), and I am always keen on learning new technology and sharing it with the team in a timely manner. If you need an expert in Hadoop Ecosystem, Apache Spark, or data engineering technologies, feel free to reach out to me. As a Certified Cloudera Apache Hadoop Developer and Azure Cloud and AWS cloud , I am confident in my ability to deliver high-quality solutions that meet your business needs.