At a glance
Industry
Insurance
Location
New Zealand
Challenge
Suncorp NZ needed to optimize its cloud data warehouse after a legacy migration ran into excessive consumption costs and slow processes.
Products and services
Data consulting
AWS configuration
Data engineering
Performance optimization
Project management
Success Highlights
- Provisioning optimizations reduced costs by $1M per year
- Batch processes optimized to meet critical daily deadlines
- Investigating issues took days but now resolved within an hour
Our approach
Through a two-week discovery process, Cognizant worked with Suncorp NZ to assess where the data warehouse was underperforming and uncover the best approach for optimizing toward the institution’s target outcomes. The team defined three objectives—to reduce data warehouse running costs by $1M per year, to always complete overnight batch runs by 8:00 a.m. and to ensure ad hoc queries returned results in a timely manner.
The Suncorp NZ and Cognizant teams united quickly to investigate available solutions and identify the best path forward. The company’s data team found Cognizant staff were eager to look for low hanging fruit for fast incremental returns, as well as to maximize value from the overall project. Across the discovery and validation process, Cognizant identified the required optimization opportunities in the Amazon Redshift implementation that could deliver the target outcomes.
Provisioning and concurrency optimization
Cognizant found that Suncorp NZ’s Redshift cluster configuration was designed with more nodes than required across both production and non-production environments. The discovery process found that average utilization was well-below capacity under this design. Adjustments to reduce nodes in the cluster could quickly improve cost without compromising performance. In non-production clusters, there was also an opportunity to further reduce requirements outside business hours to maximize cost savings.
To optimize batch processing, Cognizant identified that many of the SQL scripts involved were executing sequentially. This created a bottleneck that a move to parallel execution would resolve by giving the warehouse the opportunity to take advantage of concurrency scaling, thus reducing the overall run times.
The project hit a difficult roadblock during one stage of the development process that required a change in strategy to achieve some of the agreed outcomes. With open and honest communication, built on the trust formed through an outcome-committed contract and “one team” approach to working, the team explored alternative options carefully and uncovered a solution that would not only work but would also deliver additional final value.
“The team felt as though it was completely joined,” says Meghan Rapson, Data Engineering Chapter Lead at Suncorp NZ. “That relationship meant that when there were pivotal moments and issues to resolve everyone bedded down together to get it done. We were like a think tank. Everything was on the table and ideas flowed to get it done.”
The overnight batch process improvements were achieved using CTRL-M optimizations that improved folder performance. This typically saved half an hour in execution time across each batch of jobs for significant overall time savings—a crucial step to bring the total runtime back into line with the 8:00 a.m. deadline. Further enhancements were achieved by excluding non-NZ data where it was unnecessary and by converting Datashare to materialized views.
Business outcomes
Through changes to the Redshift cluster design, Suncorp NZ saw its AWS costs reduced by over 65% with projected savings to the organization over $1 million per year. This achieved the target cost reduction by the end of the project lifecycle. The optimized overnight batch jobs are now consistently completed between 4:00 a.m. and 5:00 a.m., well ahead of the 8:00 a.m. deadline. Every member of the Suncorp NZ team can start the day with confidence that every data point is up to date and ready for new activity when they hit their desk and meet the needs of their customers.
Clarity and commitment
With the foundation of the Suncorp NZ and Cognizant partnership built on an outcome-based, fixed-price contract, the teams knew they were united in their commitment to working as efficiently as possible to meet the clearly defined objectives. This was particularly clear when unexpected difficulties arose and new ideas were needed to deliver on the agreed outcomes. Coupled with strong communication through their one team approach, Cognizant and Suncorp NZ not only found the right path forward but achieved enhanced value in the process.
The outcome saw significant cost reductions through optimized Redshift clusters:
- Production cluster cost reduced from $67,500 to $28,100 (-58%)
- Non-production cluster cost reduced from $28,500 to $6,200 (-78%)
