SUDHEER PALYAM

• Energy Australia embarked on a journey to build a Cloud Data Platform on AWS which involved three main streams: Foundation, Customer & Markets
• I am responsible for complete Foundation Architecture, Design and overseeing Delivery of the platform using AWS for warehousing and Databricks for Machine learning and AI
• Cloud Data Foundation: Define and implement cloud-based data strategies that optimize performance, scalability, and cost-efficiency using AWS cloud services
• Creating enterprise data models for self-service analytics at all levels
• Enabled for 16 internal key business units
• Data Architecture Design: Develop end-to-end data architecture solutions that leverage AWS (Redshift, Lambda, Airflow (MWAA)) services and Databricks, data integration, transformation, storage, and analysis, information classification (RBAC), Security
• Databricks Implementation: Architect data processing pipelines, analytics workflows, and machine learning pipelines using Databricks' unified analytics platform, and ML Operationalisation
• Building data products and advanced analytics for internal and external users
• Data Governance: Establish data governance frameworks and practices for maintaining data quality, security, and compliance across AWS and Databricks environments, using Alation
• Solution Collaboration: Collaborate with enterprise architecture, security architecture, data engineers, data scientists, and business stakeholders to understand data requirements and translate them into effective architecture designs
• Data Security: Collaborate with security architects on pro-active remediation, Implement security controls and encryption mechanisms to protect sensitive data stored and processed within AWS and Databricks
• Project Management: Plan and manage data engineering projects, including setting priorities, allocating resources, and monitoring progress and measuring success
• Coach and mentor team members on technologies, techniques, and excellence in data delivery.

• This is a pivotal role in Commercial Domain Technology Architecture in delivering the
• Audience Data Hub based upon Customer Data Platform (CDP) technology as part of a major revenue generating stream program
• Supporting business stakeholders across the NewsCorp affiliates by providing technical leadership and direction in defining strategic technology solutions that meet stakeholder needs
• Working with stakeholders, both senior leadership and subject matter experts and vendor platform architects; responsible for producing the overall architecture design, assess solution options, drive experiments on Customer Data and Intelligence technologies, plan for the technology transitions, produce high-level solutions and oversee the detailed design artefacts that will be the basis for the initiative delivery
• Achievements
• Successfully Lead a multi-staged strategic transformation programs solution architecture, development and delivery on Adobe Experience Platform, one of the first CDPs in Australia
• Architected end to end platform to onboard customer demographic, behavioral, Identity data from 15 different 1st party, Affiliates and 3rd party systems under Data privacy & security compliance and perform cross-domain anonymized identification of cross-domain users using Adobe Experience Platform and ID5 technologies
• Design and deliver end to end high-level solutions that derive maximum business by increasing audience size from 3 million to 15 million
• Define Customer Data Design and modelling for consumption across the Commercial and Marketing use cases by onboarding data from Realestate.com.au, Binge, Foxtel, Streamotion & Kayo
• Contribute to multi-year data platform roadmap to enabe ML/AI use-cases.

. Designer at NAB Data Hub (NDH) one of Australis’s renowned banking standards grade date-lake on AWS. Designing and implementing mechanism to automate capture of technical metadata, augment it with Business Metadata. Using AWS Glue Catalog, Hive Event Listeners, SNS/SQS, DynamoDB, API Gateway, Neptune Graph DB, Lambda functions.

. Establish dynamic data lineage DAG from source to target systems which includes multi-tier technologies like Informatica BDM, Spark Jobs, Redshift stored procedure, Python Lambda scripts etc.

. Devised a mechanism to identify Data lineage gaps and Data Quality scoring.

Achievements

ü Automated metadata & lineage harvesting which significantly reduced data source onboarding time from days to hours, reduced human dependency & errors and achieved improved Data Governance coverage.

ü Implemented a vendor-agnostic cloud native customizable solution which helped bank to reduce vendor product licensing & services costs.

ü Hands-on in taking the solution from design to production along with fellow Data & Devops engineers under script security, compliance, agile, code driven provisioning and change managed atmosphere.

ü Automated integration of Informatica EDC (enterprise Data Catalog) & Axon based metadata capture and lineage using its APIs to data transformation in Spark jobs & ETL tools.

• Worked for clients Telstra and Australia Health building One Analytics & Cyber Security Platforms
• Established innovation and production ready Data Management Platform with combination of on-prem and cloud services
• Managed Data Ingestion pipeline implementation in Spark, Nifi & Airflow
• Achievements
• Led data Curation Pattern implementation to handle various data formats, accommodate schema changes/SCD, create unified metadata and catalog, data profiling/consistency checks
• Performed a detailed architecture review and recommendations of an existing big data environment
• End-to-end review of Platform & infrastructure, data ingestion, security, operations, data management & governance, continuous integration, applications & operational analytics use cases
• Identified data platform upgrade path, led preparation, communications, testing and upgrade execution
• Brought up platform to the latest and greatest stack - one of the few in the world
• Modernized data platform and introduced advanced capabilities like Erasure Coding
• Paved a path for potentially reducing data storage by 50% and hence saving 2M$ in platform costs
• Proposed unified data store using Data-warehouses which caters both interactive analytical queries, ETL jobs and long-running machine learning loads
• Improved BI & Analytic user experience with access to low-latency and interactive query capability from minutes to seconds
• Reduced platform & operations cost by not having to replicate data, repurpose hardware resources for critical needs
• Designed and integrated data archival into S3 tiers to reduce storage overheads for 1.8PetaBytes of data
• Enhanced Data Lake to accommodate a security operation use cases using Apache Metron, MITRE att&ck framework.

• Built the Big Data Engineering capability for NBN
• Performed a technology scan across new technologies in the market, designed experiments, implemented NBN business solutions leveraging an established, cloud-based, big data and analytics framework
• Worked with business to iterate and refine visualizations of analytical outcomes
• Applied DevOps, Continuous Integration, Delivery & Testing of ML Models to Productionisation
• Achievements
• Drove the operationalization of selected analytical models by creating portable ML Models in PMML, MLeap, SageMaker, Spark ML Pipelines
• Defined standards for data pipelines to support NBN’s analytics platform, metadata management and Created a Data Catalogue using Glue, CKAN, Apache Atlas & Apache MetaModel
• Led service discovery and deployment for Kubernetes at including running applications like Nifi, Spark Jobs as Docker images in Kubernetes, monitoring, scheduling, load balancing them
• Built server-less applications stack and response dashboards using ReactJS, NodeJS, Lamdba & Glue
• Supported prototyping, shakeout and stand-up of new analytics capabilities, implementing data wrangling and feature extraction in Spark Scala
• Conducted market study on Build vs Buy machine learning options and evaluated Databricks MlFlow, Cloudera Datascience Workbench, AWS Sagemaker, AWS docker based model deployment
• Evaluated and proposed solution options
• Resulted in code-driven prediction to production model operationalization and serving platform, responsive to change
• Improved data sourcing, curation, feature engineering and model deployment through implementation of a common feature-store in AWS DynamoDB
• Freed up data scientists by 50%, providing immediate access to data.

• Led architect solutions with principles of Agility in the Architecture, Focus on Data Quality, Design for Scalability, Address Data Analysis, Capture and Maintain Metadata, Data Governance and Integration with existing ecosystem
• Achievements
• Created a fault-tolerant Generic Data Ingestion Framework for Real-time Data Lake (RTDL) to enable use as the primary repository for raw data ingested from various Banking data sources
• Delivered into existing Cloudera based Hadoop Big Data Platform (Enterprise Big Data) that enables Data Scientists to search a catalogue of data, derive customer insights and solve customer problems
• Designed a New Payments Platform (NPP) Data Ingestion Pipeline including ingest of NPP Notifications, Emails, Messages into Hadoop Data Lake and a SOLR Search.

• Developed a distributed, fault tolerant, scalable, streaming data analytics framework for Airplane Health Prediction and incorporate Statistical and Machine Learning algorithms in (near) real time for fault diagnosis and prognosis using Azure Stack (HDInsights, EventHub, Iot Edge/Hub, Functions)
• Designed a Portal and Data transmission workflow for Airlines to Uplink Commands and Downlink Aircraft Sensor/Log Data leveraging Cloud infrastructure
• Developed a supervisory control and data acquisition system with a graphical user interface (GUI) to connect to and diagnose various Robots and connected hardware on a factory floor.