Stress testing (also known as load testing) is one of the popular goals in prerelease testing. Yet as recent media kerfuffles with product launches such as Healthcare.gov and SimCity have shown us, they can be risky and damaging endeavors if not handled correctly. So let’s take a look at how stress testing can be done to properly help your product release, rather than hinder it.
What is Stress Testing?
A stress test is the coordination of hundreds or thousands of users to test the threshold or scalability of a product’s infrastructure. Essentially, it’s asking lots of testers to use your product at the exact same time to see if it breaks. If it breaks too soon, you know your infrastructure isn’t ready to handle the load of your users. If it doesn’t, you’re one step closer to a successful product launch.
When Stress Testing Goes Wrong
Healthcare.gov and SimCity are both excellent examples of what happens when a system isn’t sufficiently stress tested. In the case of Healthcare.gov (the federal health insurance exchange launched as part of Obamacare), the federal government didn’t properly test the system before releasing it to the public, so when thousands of people tried to use the system in its first days, it broke down. This resulted in weeks of media hubbub about the failure and speculation about the fate of the entire program.
With SimCity, the problems arose on the first day of the launch of SimCity 5, as thousands of frustrated fans were unable to download or play the game. This resulted in days full of anger and apologies as the SimCity team tried to get their server infrastructure up to snuff while the product was panned in the press. Instead of building excitement and buzz about the game, they attracted lots of negative publicity and crippled their entire launch.
In both of these cases the team responsible reportedly did some beta testing of the system beforehand. In the case of Healthcare.gov, the problem was that most of the testing didn’t happen until days before the launch and focused on pieces of the website, rather than the whole. This meant the complete system wasn’t thoroughly tested before the public launch. The government even reportedly did some stress testing before their deadline and the system broke after just a few hundred people tried to log in, but they launched it anyway.
With SimCity, the company reportedly beta tested with a limited version of the product, so they didn’t get a realistic idea of how customers were going to use the product. When the real product was launched, it stressed the server infrastructure differently than they expected. This combined with the sheer number of people playing to break the system.
How To Run a Successful Stress Test
Both of these examples show not only the importance of stress testing a system before it’s launched to the masses, but also the importance of stress testing it correctly.
- Test in stages. If you think your infrastructure can handle 5,000 users at once, start with 500. If that works, scale to 1,000, then 2,500, and so forth. Often systems break much earlier than you expect them to. It’s better to start small and build to find your breaking point instead of asking 5,000 testers to log in at once only to find out that the system breaks at 200. If your system does break early (like the Healthcare.gov site did), fix the glitches and start the staged testing process over again, repeating your tests until you reach the load you need to have a reliable system.
- Don’t combine it with your marketing-focused public beta. If you’re planning to run a public beta test, it may be tempting to combine the tests since you’ll have hundreds or more beta testers in line for your public beta. The problem is that many public betas are designed to reach marketing goals, such as getting attention for the product from the media and the public before the official launch. If your product breaks after only a tenth of those excited users have logged on, you can be sure that most of the initial chatter about your product is going to be negative. Instead, make sure that you appropriately stress test your system before the marketing beta, so all the attention will be on how well your product works.
- Test using the complete version of your product. If you’re trying to test a system, you need to mimic real usage of the system as much as possible. If you’re using an incomplete or stripped down version of your product to do the testing (like the SimCity team did), you’re not getting an accurate picture of the stresses your system will be under.
- Above all: do one! As the above examples show, not properly testing your infrastructure before it’s thrust into the public eye can have disastrous consequences. Your initial launch can make or break your product, so make sure to test your system thoroughly so you know it can handle the excitement.
Stress testing is an often overlooked part of prerelease testing. With the proper strategy and timing, however, it can help you avoid a PR disaster. For more on using public betas as part of your launch strategy, download our whitepaper. If you’re interested in finding out more about how Centercode can help you run a stress test, request a demo.
