Data Engineer with 6+ years of technical expertise in data engineering and modern data architecture to design and develop efficient and scalable data solutions in various domains that includes Financial, IT security, High Performance Computing and Ad. Tech • Design and develop data pipelines, data processing applications in Apache Spark, Python, Scala, and Hive with Hadoop Ecosystem on AWS cloud, Kubernetes and Hadoop distributions • Expertise in optimizing and scaling data solutions for large-scale data sets with deep understanding of big data processing frameworks Spark, Hadoop, Hive, EMR, Athena, Glue • Adept at cloud-based data engineering, utilizing services in the AWS ecosystem such as EMR, Athena, Glue, S3, MWAA, Lambda Functions, CloudWatch, RDS, EC2, ECS and IAM • Experienced in developing Workflows for deploying and managing data solutions orchestration using Airflow and AWS Step Function • Extensive experience in working with distributed databases like Hive, Kudu and Impala • Efficient in writing real-time processing pipelines using Spark Structured Streaming with Kafka • Proficient with various file formats (Parquet, Avro, JSON, CSV, ORC) and compression codecs (GZIP, Snappy, LZO) Skills: Spark, Scala, Python, SQL, Kafka, Hive, Impala, Kudu, Sqoop, Hadoop, HDFS, YARN, Airflow, Databricks, AWS Cloud (EMR, Athena, Glue, Lambda Functions, Step Function, CloudWatch, S3, RDS, MWAA, EC2, ECS etc), ELK Stack (Elasticsearch, Logstash, and Kibana), MySQL, Docker, Kubernetes, CircleCI, GitHub, BitBucket, Bash Scripting