Business Need:
Aviation Industry has a wide range of Aircraft engines and has very huge data set to be processed on day-to-day basis. EM stands for engine modernization for Aerospace. Aerospace’s Engines business is experiencing unprecedented ramps in production for new engines, including LEAP. EM have strategic priority to modernize processes while replacing legacy systems by creating single source of truth. While ingesting data to Aurora, there was significant ingestion latency and function execution was taking more time. Also, Some Cybersecurity Maturity Model Certification (CMMC) controls are more sensible to implement in AWS Redshift.
Solution and Approach:
AWS Redshift comes with high reliability and high availability, along with the best performance. As part of the solution, ingested all the source data in redshift and migrated all functions and reports from Aurora RDS to redshift and observed significant performance improvement. Fine tune of the code helped as some functionalities are different in redshift before migrating the functions. Implementation of CICD to avoid manual deployments and integrate the system with single sign on tool.
Below is the high level architecture of the solution implemented.
Benefits:
AWS Redshift is very high reliability and high availability of ~99.99%
Function runtime reduced from 2.5 hours (aurora) to ~ 8-10 mins (redshift)
Data latency dropped and data is available near real time.
Automated deployment by CICD (using multiple AWS components)
Very minimal failures in the functions compared to Greenplum.
Migration Volume:
Talend Jobs: ~150
HVR Jobs: ~240
800+ Functions
200 Spotfire reports
Performance Improvement: >50%
Reliability and Availability: 99.99%
