Testing in Production

What is testing in production?

Testing in production (TIP) is a development practice in continuous delivery that involves deploying new code or updates into a production environment where it is tested with live users. Using live production environments instead of development, testing, staging, or pre-production environments not only ensures you have the most realistic environment for testing—but, paired with other progressive delivery techniques, it reduces the risks of a bad release by limiting exposure to a small percentage of users.

Why is testing in production important?

You can’t expect internal testing and QA teams to uncover every issue. Of the hundreds of beta programs run each year by Centercode, 100% have bugs and roughly 75% have critical severity issues found by testers. Encountering issues at beta or later is basically a certainty.

Now, given enough time and resources, you can certainly improve the amount of issues you find with pre-release testing. But at that point you’re slowing down your release cycles and chasing diminishing returns. For many organizations, it makes more sense to test limited releases in production and get feedback and performance data from real-life usage sooner (and more frequently).

How teams perform testing in production

Organizations can leverage several techniques  in order to maximize the impact and minimize the risks of testing in production:

1. Delta testing: Recruiting a group of users to receive ongoing, early updates and provide feedback. This is a replacement for traditional beta testing.
2. Feature flagging: Using a toggle to enable/disable features for a subset of users. In this case, your delta testers would be the first group to receive the feature, but there can still be others.‍
3. Traffic splitting: Diverting a small percentage of users to a new release, while the previous release is still served to the rest, as in canary deployments. This acts as a hedge against issues that might’ve been missed in QA and/or user testing.