Once your beta test is rolling, you’ll need to focus on getting your testers interacting with your product. Here are nine tips to help you build momentum and get your testers pointed in the right direction.
1. Use the Right Tool for the Job
There are many types of feedback mechanisms. Beyond bug reports and feature suggestions, other common activities in betas include the use of surveys, tasks, daily journals, discussion forums, wikis, etc. The key is to think about your goals and select your tools based on what you’re trying to accomplish. For example, if you know that documentation feedback is critical, that’s a great opportunity to enter your docs into a wiki and crowdsource improvements.
2. Encourage Participation with Broad Tasks
Some testers lack the initial drive to independently explore your product and report back their findings. We’ve found that giving people a set of very basic, general tasks will help kick-start their use of the product, after which they’re more likely to do their own exploration. Note that these should not include tasks that will focus the tester on very specific features or activities, but rather the product as a whole (i.e., download the software; load the software; review the online help). In most cases, while you may have to nurture participation in the beginning, testers will be much more independent once they build some momentum.
3. Use Specific Tasks to Regress Fixes
One area where a diverse and reliable tester team really shines is in regression testing. If you’ve fixed some known bugs, verify that you’ve solved the problem with a group (or, in some cases, all) of your testers. You can segment your team by test platforms that were known to exhibit the bug and assign tasks that follow the specific steps required to recreate the issue. Or, you can set your entire team after the problem just to make sure it’s really gone. The added benefit of this is that testers will experience the results of their efforts firsthand, leading to increased participation.
4. Assign Objectives, Not Steps
“Goal-based test scripts are a much better way to get a thorough assessment of your software’s usability. If you give them a task like ‘Turn on the Scheduler,’ you not only assess how the Scheduler works, but how easy it is to find and use it.” — Tony Weiss, Symantec
5. Give Reasonable Deadlines
Remember, the people volunteering to test your product are (usually) doing this in their free time. It’s important to respect that and give them a real opportunity to finish the activities (surveys, tasks, etc.) assigned to them. We find that 2-3 days is reasonable for relatively simple tasks, while a week is appropriate for more complex assignments. You can opt for shorter deadlines when necessary (and only sparingly), but understand that completion rates will probably suffer for it.
6. Issue Surveys Occasionally
Try to limit surveys to about one or two per week. They’re an incredibly useful tool, but there are consequences to assigning too many. Frequent assignments discourage your participants from exploring the product on their own, and if taken to the extreme, they will quickly frustrate and burn out your testers.
7. Avoid Testing Failures and Lost Causes
“Don’t ask testers to perform a test you know will fail. This just causes unneeded frustration. Similarly, don’t ask your testers a question unless you can act on the feedback. Testers want to make a difference (which is why they volunteered to test), so if their feedback goes unheard, they will be upset and will be less likely to volunteer for you in the future.”
— Amanda Dawson, TiVo
8. Limit Your Survey Questions
When testers are presented with a long survey, they may decide to skip it just on looks alone. It’s not necessarily important whether the questions are simple or complex. Thus, we recommend limiting surveys to between 5 and 15 questions. An even better approach is to only display the most pertinent questions by making them conditional on the answers to earlier questions (a feature found in our software). Not only does this give you the opportunity to craft a more detailed survey, but it also makes the survey appear less daunting at first glance. For more tips on surveys, check out: 10 Tips for Writing Great Beta Surveys.
9. Time Assignments Strategically
If you’re planning a long beta, it can be a helpful to time your tasks and surveys so they occur in the later stages of testing. Early on, many testers are likely to be excited and focused on exploring, and will not need any additional push. However, as the test goes on, the extra direction that assignments provide can often breathe new life into a waning beta. Tasks make it very clear what you’re looking for from the tester, while surveys tend to be an easy method of giving feedback (particularly when they consist mostly of drop-downs and check boxes).
Hopefully these tips will help you build momentum as you start your beta test. If you’d like more information on keeping your testers engaged and involved, check out Reaching 90% Beta Test Participation.
