 High Stakes System Optimization
Client Overview
A large national mortgage lender that originates and services home mortgage loans through 280 branch offices and over 14,000 affiliates across the United States.
Business Needs and Project Objectives
Client was a leading subprime mortgage lender sponsoring the Super bowl 2005 half time show for $15 million. In addition, they had purchased advertising during the show highlighting their mortgage offerings.
As a result, the client was expecting a large number of web hits on their client facing website. Given the fact that the 2004 half time show sponsor's website had crashed due to increased load, the client was very eager to optimize their website to be prepared for the expected load.
Challenges / Requirements:
The client wanted sub second response time on all of the pages on the website at 5X the load on their previous load peaks. The site had to be ready to handle 170,000 unique visitors in a span of 2 hours. Initial testing on the site indicated marked client site delays around 50,000 visitors and heavy delays around 70,000 unique visitors.
In addition, AAXIS Group had to coordinate the effort with external content providers and bandwidth providers to ensure that the testing was complete and represented the actual client experience.
This had to be performed under a tight timeline of 3 weeks and was to be executed before the Super Bowl Sunday
Role of AAXIS:
A simple approach was to throw more hardware at the problem. However, AAXIS Group realized that the solution needed to be more holistic. We deployed a consolidated approach with monitoring of various layers including network, database, hardware, operating systems, web server and application layers.
By using proprietary process known as AAXIS Resolution Methodology (ARM), we were able to bring together disparate teams of technology teams to perform site testing in a unified manner.
AAXIS System Optimization Services (SOS) engaged and led the overall testing strategy.
Additionally, we provided the technical expertise to support the effort.
AAXIS:
- Performed a project gap and risk analysis
- Designed and coordinated a performance testing strategy that could be executed within project timelines
- Coordinated the preparation of a high-scale testing environment
- Performed in-depth architectural review and validation
- Identified and helped resolve scalability and fault-tolerance issues with network infrastructure
- Identified load balancing defect and tested configurations to optimize load distribution
- Executed local failover testing for all redundant application tiers
- Performed global cross-site failover testing from primary datacenter to remote disaster-recovery site
- Hardened all tiers of the production site: web, application, database, and network
- Coordinated and supported the production rollout and transition strategies
Listed is a quick summary of changes that were recommended
- Optimization of Weblogic application server settings (handling prequels)
- Optimization of threads and memory mapping on the web server layer
- Increased bandwidth on incoming pipe
- Increase hardware on image servers
- Recommended replacement of faulty secondary load balancer hardware
- Optimization of SQL Query execution by recommending indexes on key tables
Results:
Judiciously recommending hardware, software, database and network settings, AAXIS Group was able to improve the overall performance on the website to handle the expected load times during the Super Bowl half time show.
From a load perspective, Super Bowl Sunday produced 3X spike in traffic. The number of visitors was up to 100,247. The site handled the loads gracefully and responded with the business SLA specified.
The site produced 3.3% conversion rate, the highest for the month. That combined with the higher load resulted in top sales for the company.
Client Quotes:
Head of Direct to Consumer Division: "Awesome! Awesome! Awesome! Congratulations everyone!"
Client Project Manager: "Consumer Direct Team - So far, it's a success! Thanks for all your dedication!"
Engineering team: "we saw approximately a 3X spike in traffic to the site after the Super Bowl, from around 6pm to 9pm. The site handled this spike without any problem; response times from the web servers remained constant throughout the day. Prequals were also higher today. We had a 3.3% conversion rate, which is the highest of the month so far.
Visitors were at 100,247 yesterday, the highest for the last three months. That's 2.4 times more visitors than yesterday. The site held steady throughout the day.
The Super Bowl microsite was the most popular visit, with over 40,000 visits, other than the home page and prequal form." |
