Product Development

Conducting Continuous User Research with Delta Testing

What is continuous user research, exactly? We like this definition from Tomer Sharon, head of user research at Goldman Sachs:

"Continuous user research is fast-rhythm research that is open-ended in nature. It's not dedicated to any specific topic, and it's not research that anyone asks for."

Unpacking that a bit, we see three core elements at work:

  • It's fast-paced. Your customers' wants and needs are constantly evolving. In order to know where they're at and use those insights in a timely manner, continuous user research has to happen quickly.
  • It's ongoing. Thanks to your customers and the point above, your product is also evolving constantly. If you want a flow of relevant and useful user research, you'll need a watertight system for collecting and delivering those insights on a regular and continuous basis.
  • It's proactive. Most of the metrics you're using to assess customer satisfaction are reactive, meaning fixable issues are getting back to you via support tickets well after launch or release. Instead of waiting for those fixes to roll in, conducting continuous user research can identify points of friction before they hit your larger audience.

You might be sitting here thinking that this all sounds great in theory, but you don't have a team to get this off the ground at your company. And that's a fair point: Not everyone has the resources to roll out a full-on continuous user research program with dedicated researchers, or the time to do the work yourself between tight release schedules. Most of us are already overloaded, after all.

Let's be clear: The world runs on data and that's what makes user research (and professional user researchers) so valuable. But while you can't replace the value a user researcher brings, you don't have to sit on your hands waiting for your ship to come in.

In fact, you can net many of the benefits of continuous user research using a process that, if you're like 70% of companies, you're already practicing: beta testing. Not just any old beta testing though. We're talking about automated user testing, also known as delta testing.

If the idea of deepening your knowledge of the user experience and reacting more quickly when issues arise makes you rub your hands together in anticipation, you're in the right place. We'll show you how to get started conducting continuous user research using delta testing, and how it delivers powerful insights even when you've got time and cost restraints.

Continuous User Research With Delta Testing

Like beta testing, delta testing gives you insight into how your customers are using and living with your product in the real world, over time. By submitting feedback, testers surface bugs and generate ideas to improve the user experience. You can then turn this feedback around and use it to increase the usability and satisfaction of your products.

But unlike beta testing, delta testing is automated. This eliminates the major time-sinks of beta testing, like organizing the feedback into distinct and usable categories, prioritizing that feedback by impact, and reporting the findings to your stakeholders.

When you line up the strengths of delta testing with the core concepts of continuous user research, you'll find a lot of overlap and problem-solving.

  • Delta testing is agile-friendly. Instead of testing across an entire product every once in a while (traditional beta), delta testing enables you to roll through smaller, more specific product areas ahead of what you're releasing in a given sprint. This allows your testers to focus exclusively on the areas that you need to validate immediately and enables you to secure the necessary deliverables to get your release out the door.
  • Delta testing runs for as long as your product does. A combination of automation and the use of a tester community allows you to use delta testing continuously, rather from project to project. This makes it a perfect match for the continuous user research you need for today's iterative products. As long as you continue to release and improve new features, you can rely on an ongoing cycle of interested, engaged delta testers to give you real-world feedback.
  • Delta testing leaves room for proactive, unguided, ongoing product exploration. During the "inactive" test periods between big releases, testers are encouraged (via automated messaging) to explore the product and submit feedback. Whenever they run into friction, get an idea, or stumble upon something they really like, they can submit feedback — and you can benefit from whatever turns up.

Getting Started

Ready to stop imagining the possibilities of continuous user research and dive in? Learn how to leverage its powerful benefits — and the other time-saving superpowers you get by adopting user testing automation — in this beginner-friendly ebook, The Delta Testing Quickstart Guide.