Is adding more testers always the best solution when you're experiencing a dip in feedback? In this Ask the Expert series, Centercode’s Chris Rader discusses large tester teams: when they work, when they don’t, and how to manage when your stakeholders want to keep adding testers.
Got a question you’d like to see on the Centercode Blog? Send an email to help@centercode.com with the subject line, “Ask the Expert”!
Managing Large Tester Teams Effectively
Hey Experts,
I’m a test manager at a company that creates mobile apps. I started running beta tests as part of my job last year, and when quarantine began, my company needed someone full-time, so I stepped up. 🙂
Needless to say, I’ve had to learn a lot quickly. Even though I feel like I’ve made significant progress, I’m still struggling to get quality feedback from projects with 500+ testers.
Whenever feedback is lacking, my stakeholders suggest adding more testers. While this does increase feedback, it becomes harder to extract the detailed insights I need to forward to Engineering. I spend so much time following up with testers that it cuts into my time for triaging and building reports. Overall, I feel like the project quality diminishes as more testers are added.
What should I do?
Thanks!
Liz R
Meet Today’s Expert
Chris Rader
Director of Product
Chris Rader is a product director at Centercode. He collaborates with our customers to interpret their beta test data, conduct market and user research, and dream up creative product solutions for the Centercode team. His superpower is the ability to paint a picture with customer data that explains the why and how behind complex problems.
Hey LR,
Happy to shed some light on this topic!
Adding more testers is a common solution to needing more feedback or survey responses, especially when suggested by stakeholders who aren't directly involved in the project.
The Pros and Cons of Expanding Tester Teams
A larger tester pool has its benefits. It increases the quantity of feedback, though not necessarily the participation rate. It’s also beneficial for gathering product analytics in mobile apps. However, if your goal is to enhance product quality, identifying actionable feedback becomes more challenging and time-consuming with larger tester teams.
Imagine there’s a crowd of people (socially distanced and masked, of course), and you want all of them to download your app. If there are 50 people, you can provide support and communicate effectively. But with 500, it’s much harder to keep everyone aligned. The same applies to your test project.
For most tests, more testers = more time. Managing more testers means more time spent getting everyone to download the app, respond to feedback, and communicate weekly activities. This can lead to pulling in more feedback, but identifying what’s actionable may suffer, making the results less valuable overall.
Making the Right Decision
Before deciding, consider your goals. If your test aims to improve specific areas of your product, adding more testers may not be beneficial in the long run. The best tactic for obtaining actionable, high-quality feedback is to focus on how you communicate with your testers.
Testers respond well to frequent, personal communication. Setting clear expectations, responding to feedback, and engaging with testers (in forums, for example) are all effective ways to increase feedback and responses in areas important to you and your stakeholders.
Having difficulty engaging your testers? Learn how to navigate roadblocks with the comprehensive best practices and troubleshooting guidelines in the Tester Engagement Pocket Map.
If your stakeholders are mainly concerned with gathering product analytics and validating popular issues, adding more testers might be the best approach. Since mobile apps aren’t limited by constraints like the number of beta units available, this shouldn’t be a concern.
However, in your case, you might need to focus your project on specific priorities (instead of trying to tackle everything) to collect and maintain quality feedback. You can achieve this by reducing the number of objectives or centering your surveys and weekly topics on your stakeholders’ top priorities. Combine this with effective communication with your testers, and you’ll get the insights you need regarding your product priorities.
And of course, the Centercode Platform manages all that feedback without sacrificing quality. It automates tasks like separating issues from feature requests, minimizing duplicates with voting, and prioritizing feedback by severity and popularity, saving you tons of time. If you want to learn more about automating feedback management, don’t hesitate to schedule a demo!
Hope this helps!
Happy Testing,
Chris
Finding Your Ideal Test Size
So, how many testers do you need to achieve product coverage and statistical significance? Find the ideal test size for your current or next project with the Test Size Calculator. This free resource will quickly and accurately calculate the number of testers you need based on your specific project. Go on, try it out.
