Services
Video meeting . 60 mins
Data Engineering Roadmap
Consultation - Become data engineer
About me
➢ Over 16 years of experience as a Data Engineer, specializing in BI migration and data warehouse projects, providing a deep foundation in IT.
➢ More than 5 years of leadership experience managing data engineering teams.
➢ 5+ years of expertise in cloud data engineering, focusing on AZURE/AWS services.
➢ Collaborated extensively with internal teams, including data engineering and product teams, to lead solution design and proof-of-concept discussions.
➢ Adept at partnering with product owners, client managers, senior leadership, business analysts, and architects to drive project success.
➢ Proven ability in providing both technical and leadership support, fostering collaboration and a positive team culture.
➢ Strong skills in RDBMS, database design, and dimensional modeling.
➢ 5+ years of experience designing azure solutions with services such as Azure Synapse, SQL Database, NoSQL, Databricks, data lake, delta tables, blob storage, key vault, and PySpark.
➢ Proficient in integrating external data sources using Databricks parquet, CSV, azure synapse and analytics
➢ Skilled in connecting storage solutions like Azure Data Lake Storage Gen2, Blob Storage, and Amazon S3.
➢ Experience in building REST API solutions and data applications using FastAPI, python, streamlit, and a wide range of libraries (requests, pandas, numpy, JSON, Azure Data Lake, Blob Storage, etc.) to support business operations.
➢ Designed and implemented security measures for REST APIs, including authentication and authorization protocols.
➢ Expertise in designing reusable solutions based on medallion architecture, leveraging azure data factory, Databricks, delta tables and delta live tables.
➢ Implemented unit testing using the Python library pytest.
➢ Experience in developing ETL/ELT pipelines with SSIS and Azure Data Factory.
➢ In-depth knowledge of SQL Server, including T-SQL and query optimization.
➢ Hands-on experience with CI/CD pipelines in Azure DevOps and agile delivery methodologies.
➢ Proficient in visualization tools like QlikView, Power BI, and SSRS.
CORE TECHNICAL SKILLS
➢ MS SQL Server, Oracle, Azure Synapse, NoSQL, Databricks, Delta tables, Delta live tables
➢ SQL Server Management, SQL Developer, Azure Storage Explorer
➢ Azure Data Lake Gen2, Azure Blob Storage, Amazon S3, Key vault
➢ Azure Databricks Workflow Management and APIs
➢ Data Warehousing, Database Modeling
➢ SQL, Python, PySpark, FastAPI, Streamlit
➢ Azure Data Factory, SSIS, Informatica
➢ QlikView, Power BI, SSRS
➢ DevOps, TFS and SVN
➢ Postman
➢ Pytest