Summary
Overview
Work History
Education
Skills
Certification
Timeline
Generic

Venkata Naveen Nandigam

Melbourne

Summary

With over 7.6 years of experience as an Azure Data Engineer, I have developed expertise in Azure Cloud Engineering, Cloud Computing, Bigdata Analytics, Data Warehousing, and creating impactful dashboards using Power BI. Throughout my career, I have honed my skills in these areas, enabling me to proficiently design and implement data solutions on the Azure platform. My passion lies in leveraging cloud technologies to optimize data processing and analysis, ultimately driving actionable insights for organizations.

Overview

8
8
years of professional experience
1
1
Certification

Work History

Azure Data Engineer

HCL
11.2021 - Current
  • Writing Pyspark code as per the business requirement in PyCharm tool
  • Calling Spark code by using azure data factory
  • Creating ADF Pipelines and datasets to process the data from source to target
  • Creating schedule and tumbling triggers and assigning to Pipelines based the frequency of load
  • Creates spark data frames to pull the data from upstream and process them through spark logic and store the final output in hive tables
  • Creates billing files through spark Scala code and share it with business for monthly billing
  • Load the data from sales force to Synapse and Adls-Hive using Pyspark code
  • Pushing the code from git local to bitbucket and raise the pull request
  • Collaborated on ETL (Extract, Transform, Load) tasks, maintaining data integrity and verifying pipeline stability.
  • Fine-tuned query performance and optimized database structures for faster, more accurate data retrieval and reporting.
  • Enhanced data quality by performing thorough cleaning, validation, and transformation tasks.
  • Encrypting the files by using gpg tools
  • Creating power BI dataflows, datasets and reports

Big Data Developer

Legato
12.2020 - 10.2021
  • Writing the business logic in spark Scala to identify the valid claims which we receive from varies sources
  • EFX the files which we receive for each much though spark Scala code
  • Analysis the business logic and made the code changes in spark Scala in case of any business logic change
  • Big Data Developer at Anthem health inc(Health Care)
  • Increased accuracy in data analysis by creating robust, reusable ETL pipelines to clean and transform raw data into usable formats.
  • Participated in code review processes to maintain high levels of quality and consistency in software development practices, resulting in reduced defect rates and improved team collaboration.
  • Contributed to the continuous improvement of big data infrastructure by monitoring system performance, identifying bottlenecks, and implementing necessary optimizations.

Big Data Developer

TCS
02.2017 - 12.2020
  • Worked on data migration project from Datawarehouse to bigdata platform
  • Loading the data from fixed and delimited files to hive tables by using spark
  • Applying transformation on hive table and loading the final output to another hive table
  • Copying data from Big Data to RDBMS by using Sqoop Export
  • Have implemented Data quality report generation every day to validate the Data flow
  • Have written a Unix script to run Spark jobs from the shell
  • Have prepared AutoSys Jill to schedule shell commands in the scheduling tool
  • Hands on experience on hive partitions and bucketing
  • Implementing logic for processing hdfs files and RDBMS tables by using Sqoop and spark
  • Experienced on loading different kind of files from HDFS to hive tables Import and export the data from various sources like (RDBMS)
  • Create python/spark jobs for data transformation and aggregation

Education

Bachelor of Technology -

Jntu Kakinada University
India
04.2014

Skills

  • Azure Data Factory
  • Azure Synapse
  • Azure data lake
  • Data Bricks
  • Power BI
  • Spark
  • Python
  • Hive
  • Sqoop
  • Sql/pl sql
  • RDBMS
  • SQL
  • UNIX shell scripting
  • IBM db2
  • Oracle
  • Teradata

Certification

  • DP-203(Azure Data Engineer)

Timeline

Azure Data Engineer

HCL
11.2021 - Current

Big Data Developer

Legato
12.2020 - 10.2021

Big Data Developer

TCS
02.2017 - 12.2020

Bachelor of Technology -

Jntu Kakinada University

Similar Profiles

CREATE PROFILE
Venkata Naveen Nandigam