Apps are both readily accessible and everywhere. Go into a grocery store, a mall, a transit station, and you’ll see apps for everything from shopping to parking to public transportation. Like me, you’ve probably installed dozens, if not hundreds of apps on your phone or tablet over the last few years. Even my elementary school-age kids are downloading and deleting apps every single day! This is what makes app Beta Testing so important.
App Beta Testing follows a similar process to validating other products with real customers, but there are a few key differences. Let’s take a closer look into some of the factors that make app Beta Testing unique, and a few key points on how to make sure your tests – and your next app – are wildly successful.
- Consider the Whole Product Experience
- Match Your Market’s Technographics
- Waterproof Your Distribution Plan
- Hold Off Pushing Updates
- Use the Crash Analysis
1) Consider the Whole Product Experience
I would be remiss to come into a conversation about testing apps without calling out the fact that most connected hardware devices have a custom app. In my own home, our speakers, lights, cameras, HVAC system, and even my jogging headphones all have an app. These provide anything from basic tweaks in functionality (headphones) to being the only way to use the device (camera). But in each case, it’s a part of the entire product experience.
If your app is part of a larger ecosystem, make sure that you account for all of the moving parts when you plan what needs to be tested on it.
2) Match Your Market’s Technographics
On average, people have six connected devices in their homes. Testing in real homes is a perfect opportunity to see how your product performs in highly varied technical environments. But you’ll run into problems if even one of these other devices conflicts with yours.
When confronted with a problem while adding a new device to their existing setup, only one in three people experiencing app problems will investigate other devices as the cause of the problem. Two out of three customers will either blame the new device or themselves.
To counter this, test your product in a spread of unique home environments. Get a range of variation in routers, smart devices, and phones to test as many scenarios as possible. You might have to look outside of your normal testing base if they have too many similarities.
Bonus Tip: Give Your Testers the Support They Need
When it comes to conserving tester energy, make sure that you are able to help out testers rapidly once you start your test. Remember that you’re working in a marketplace inundated with apps. If a tester runs into an issue and doesn’t get immediate support, they’re likely to deal with your app the way they normally would – by removing it. That’s one less tester and environment you’ll have the chance to validate.
3) Waterproof Your Distribution Plan
Distributing pre-release apps is not as straightforward as just sending a piece of software. Correct app distribution will require some additional systems and some time to set them up and maintain them. It will also require instructions for testers.
It’s critical to map out the tools you’ll be using and to identify any limitations that could impact tester onboarding. Some organizations have predefined tools for distributing their apps, but their use could be limited to internal distribution.
Finally, when testing with hundreds of testers, be aware that you’ll have to spend a good deal of time troubleshooting app installation issues. Even though these distribution tools continue to advance, they still require testers to go well beyond their normal usage and have to be set up correctly in order to work.
Here are a couple of the more common app management platforms.
4) Hold Off Pushing Updates
You’re likely going to get updates for your app while Beta Testing. Although it might be tempting to push each update out to your testers as soon as they’re available, this is often not the best idea for two reasons.
First, you need to make sure that the update is stable. Even though you have testers who really want to help you out, everyone has a limit to how many issues they can run into before they throw in the towel.
Second, you might be muddying your feedback. To know your testers’ perception of your product, you need to let the experiment run its course without interference. Multiple changes during app Beta Testing warp the data on your testers’ perceptions. Updates make feedback inconsistent from the beginning of the project to the end, leading to inconclusive results.
5) Use the Crash Analysis
If you use one of the distribution platforms I listed in Tip #3, you are going to have access to varying levels of device information that can give you valuable insight into usage and performance. Tying this information back into your user feedback will give a holistic view of your application.
For instance, TestFairy lets testers record short videos of interactions and make annotations. The program sends this data back to you along with the technical details of the device performance at that time, like processor utilization and data rates. Combined with your standard Customer Validation feedback, it paints a full picture of user satisfaction. From here, you have the in-depth knowledge you need to improve customer experience.
More Resources for App Beta Testing
If you are running beta tests for your app, or plan to in the near future, we have a test planning kit that will get you moving in the right direction. Download the Mobile App Beta Testing Kit for free templates and a step-by-step guide to getting the jump on your beta test plan!
