Harikrishna Koti
Sydney,NSW
Summary
Highly skilled Azure Data Engineer with 10 years of experience in designing, implementing, and managing data solutions on the Microsoft Azure platform. Certified data engineer and programmer proficient in building scalable and efficient data pipelines, performing data transformations, and ensuring data quality. Expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Blob storage, and Azure SQL Database. Proficient in writing complex SQL queries and leveraging big data technologies like Apache Spark (PySpark) for data processing. Skilled in data modeling, schema design, and performance tuning for optimal data analysis. In-depth knowledge of various data engineering concepts, tools, technologies, and ETL tools like Informatica Power Centre. Experienced in Agile environments, managing project timelines, and effectively prioritizing tasks to meet project goals.
Overview
11
11
years of professional experience
1
1
Certification
Work History
Data Engineer
ING Australia
04.2023 - Current
- Optimal extraction of data using Azure Data Factory from different sources and maintaining Data in Data Lake services
- Converting Business logics using Azure Databricks using PySpark writing to Delta formatted tables to ADLS locations
- Optimizing the long running Azure Data Factory pipelines and making them efficient
- Performing Data cleansing, enrichment and transformation operations on ingested data, and automation of same
- Built Optimized pipelines which gets data from On-premises to Azure Data lake storages
- Using Apache Spark for Distributed Data processing, and integrated services like ADF, Synapse Analytics
- Created pipelines in ADF using LinkedServices/Datasets/Integration Runtime to perform ETL actions
- Developed Spark apps using PySpark, and Spark-SQL for data transformation and aggregation from multiple file formats
- Experienced in performance tuning of spark applications for setting right batch interval time
- Creating build and release for multiple projects in Dev/UAT/Prod environments using Visual Studio Code
- Helping Junior resources to bring them up to the speed
- Converting Semi-structured formats to Structured formats
- Knowledge on Data governance, security principles and experience with Azure Security centre and Azure Active Directory
- Responsible for identifying the best distribution techniques in Azure Synapse Analytics for different types of tables.
- Worked effectively in fast-paced environments.
- Self-motivated, with a strong sense of personal responsibility.
Lead Software Engineer
Wells Fargo
05.2013 - 03.2023
- Created data pipelines in ADF to copy the data from on-prem to Azure blob storage
- Strong understanding of data modeling and ETL (extract, transform, load) processes
- Data migration from On-prem to cloud, created self-hosted integration run time to copy the files
- Created the datasets and built generalized/parameterized pipelines
- Followed the Medallion architecture which was proposed by Microsoft as well as data bricks
- Implemented Azure data bricks clusters, notebooks, jobs and auto scaling
- Deployments using the Resource Manager templates
- Created optimal data pipelines, cleansed, mapped and transformed as per business and technical requirements
- Provided technical leadership and thought leadership as a senior member of the Analytics Practice in areas such as data access & ingestion, data processing, data integration, data modelling, database design & implementation
- Knowledgeable working on Informatica Power Centre
- Analyze the data based on the requirement and discuss with business the different scenarios to consider in order building an efficient product
- Provide the development efforts for each detailed requirement during the Agile scrum planning session
- Providing the custom views to reporting team
- Reduced turnaround time of reports generations by 30%
- Developed optimized ETL mappings
- Operationalize data ingestion, data transformation and data visualization for enterprise use
- Troubleshoot job failures, data ingestion issues, or performance bottlenecks to ensure smooth operations
- Document workflows, processes, and solutions to facilitate knowledge sharing and maintain system documentation.
Education
Master of Science - Computing and Information Systems
Liverpool John Moores University
Liverpool, UK01.2011
Bachelor of Technology - Information & Technology
Jawaharlal Nehru Technological University
01.2009
Skills
- Azure Data services
- Informatica Power center 10x/14x/15X
- Scheduling tools AUTOSYS, CONTROL-M
- Shell Scripting
- Data Integration and Transformation
- Big Data Technologies
- Cloud Infrastructure
- SQL Expertise
- ETL development
- Problem Solving and Troubleshooting
- Continuous Learning and Certifications
- PL and SQL Development
- Oracle SQL Developer
- Oracle Performance Tuning
Certification
- Microsoft Certified: Azure Data Fundamentals, DP-900, https://www.credly.com/badges/f456a8c2-12f3-4402-bdf4-817ee88dc653
- Microsoft Certified: Azure Fundamentals, AZ-900, https://www.credly.com/badges/a0fe9c14-e9e2-47f4-9a74-224c77dee86d
Languages
English
Telugu
Hindi
Residency
Australian Permanent Resident
Personal Information
- Title: Azure Data Engineer
- Date of Birth: 06/12/87
- Marital Status: Married
Hobbies and Interests
- Technology Exploration
- Coding & Programming
Timeline
Data Engineer
ING Australia
04.2023 - Current
Lead Software Engineer
Wells Fargo
05.2013 - 03.2023
Master of Science - Computing and Information Systems
Liverpool John Moores University
Bachelor of Technology - Information & Technology
Jawaharlal Nehru Technological University
Similar Profiles
JEMMA SHANNONJEMMA SHANNON
Customer Care Specialist at ING AustraliaCustomer Care Specialist at ING Australia
Jydan CooperJydan Cooper
Know Your Customer (KYC) Specialist at ING AustraliaKnow Your Customer (KYC) Specialist at ING Australia
DHIREN KESWANIDHIREN KESWANI
AML Analyst (TM + CDD) at ING AustraliaAML Analyst (TM + CDD) at ING Australia