Services

Video meeting . 60 mins

Data Engineering Roadmap

Consultation - Become data engineer
800
Popular

About me

➢ Over 16 years of experience as a Data Engineer, specializing in BI migration and data warehouse projects, providing a deep foundation in IT. ➢ More than 5 years of leadership experience managing data engineering teams. ➢ 5+ years of expertise in cloud data engineering, focusing on AZURE/AWS services. ➢ Collaborated extensively with internal teams, including data engineering and product teams, to lead solution design and proof-of-concept discussions. ➢ Adept at partnering with product owners, client managers, senior leadership, business analysts, and architects to drive project success. ➢ Proven ability in providing both technical and leadership support, fostering collaboration and a positive team culture. ➢ Strong skills in RDBMS, database design, and dimensional modeling. ➢ 5+ years of experience designing azure solutions with services such as Azure Synapse, SQL Database, NoSQL, Databricks, data lake, delta tables, blob storage, key vault, and PySpark. ➢ Proficient in integrating external data sources using Databricks parquet, CSV, azure synapse and analytics ➢ Skilled in connecting storage solutions like Azure Data Lake Storage Gen2, Blob Storage, and Amazon S3. ➢ Experience in building REST API solutions and data applications using FastAPI, python, streamlit, and a wide range of libraries (requests, pandas, numpy, JSON, Azure Data Lake, Blob Storage, etc.) to support business operations. ➢ Designed and implemented security measures for REST APIs, including authentication and authorization protocols. ➢ Expertise in designing reusable solutions based on medallion architecture, leveraging azure data factory, Databricks, delta tables and delta live tables. ➢ Implemented unit testing using the Python library pytest. ➢ Experience in developing ETL/ELT pipelines with SSIS and Azure Data Factory. ➢ In-depth knowledge of SQL Server, including T-SQL and query optimization. ➢ Hands-on experience with CI/CD pipelines in Azure DevOps and agile delivery methodologies. ➢ Proficient in visualization tools like QlikView, Power BI, and SSRS. CORE TECHNICAL SKILLS ➢ MS SQL Server, Oracle, Azure Synapse, NoSQL, Databricks, Delta tables, Delta live tables ➢ SQL Server Management, SQL Developer, Azure Storage Explorer ➢ Azure Data Lake Gen2, Azure Blob Storage, Amazon S3, Key vault ➢ Azure Databricks Workflow Management and APIs ➢ Data Warehousing, Database Modeling ➢ SQL, Python, PySpark, FastAPI, Streamlit ➢ Azure Data Factory, SSIS, Informatica ➢ QlikView, Power BI, SSRS ➢ DevOps, TFS and SVN ➢ Postman ➢ Pytest