Pop quiz: does the state of your product impact tester engagement during Customer Validation projects?
As you’ve probably guessed from the title of this blog post, the answer is “absolutely.” Stopping issues, critical errors, or tons of small bugs can cause tester engagement to decrease dramatically.
“OK hold up on a second,” you might be saying to yourself. “The impact of recruitment, communication, and tools I understand very well — I’ve been reading your entire Engagement Workshop Series after all — but isn’t the whole point of customer testing to assess the performance of a buggy product?”
Yes and no. It’s one thing to recruit technical users for the specific purpose of routing out critical bugs and other performance issues. But if you’re asking everyday target market testers to perform tasks on product functions that are very error-prone or still in the works, you can expect to see a sharp participation drop-off. It’s simply too difficult and time-consuming for them to successfully get through weekly activities, let alone provide feedback.
When testers run into areas of your product that haven’t been fully developed, it also prevents them from experiencing and providing feedback about your product the way your future customers will. This takes a toll on how relevant incoming feedback is to your project goals, since tester engagement is just as much about quality as it is about quantity.
Here are the warning signs that excessive product friction is decreasing tester participation.
Friction with Your Product: Warning Signs
- Testers are unable to complete activities. There’s a difference between “unwilling” and “unable.” If frequent issues or problems with product areas prevent testers from completing their activities, you’ll see a sharp decrease in tester engagement.
- Testers aren’t submitting ideas or praise. If testers encounter too many bugs, they’ll spend the majority of their time reporting issues. They won’t be able to get the full experience of key product features in a way that allows them to provide ideas and praise.
- Product usage is low overall. If usage metrics show that target market testers aren’t using your product in a meaningful way, it could mean that your product is too buggy or inconvenient for testers to use in the ways they’d naturally want to use it.
- There is a large number of duplicated critical issues. When a significant number of testers report the same blocking issues, it’s a sign that bugs are interfering with the tester experience.
Quick Fixes for Friction with Your Product
You wouldn’t expect your customers to muddle through a frustratingly buggy product on the market. In the same way, your testers need a mostly clear product experience to thoroughly test key features and provide meaningful feedback. They need (and want!) every opportunity to give their authentic impressions and experiences.
Here are the steps you should take if you’re experiencing some of the warning signs from above.
- Check Development Progress. If testers aren’t completing activities or using your product at all, make sure your product is substantive enough for your testers to truly use. Do testers have a tangible, feature-complete product to provide feedback about, or are they testing a prototype, wireframe, or concept version? Are all of your topics and activities reasonable, given your product’s current state?
- Check Your Testing Phases. If a high number of your testers are reporting critical issues, check back on how your product performed before customer testing. Did it go through rigorous QA testing and/or dogfooding processes before it landed in the hands of customer testers? Were all of the major issues resolved during those phases? Are there open tickets for your product that could be negatively impacting the tester experience?
Your Roadmap to Increasing Tester Engagement
It’s tough to navigate all the moving parts behind maintaining tester enthusiasm and soliciting high-quality feedback. The Tester Engagement Pocket Map is full of tips and tactics for troubleshooting your CV project and comes with a compact tester journey map to point you in the right direction.
