Centercode is headed to Boston! Register for our user group on May 21st!
Book cover for "The Future of Product Metrics"

The Future of
Product Management Metrics

Introduction

There is no shortage of metrics in business. As Peter Drucker famously said, "You can't manage what you can't measure." And as a product manager, you're responsible for managing the vision and communicating the information you glean from metrics into a strategy that various teams within your organization rely on.

Metrics are essential because they help you shape the product as it develops and forecast the success of every release. If you're a seasoned product manager, you're probably already familiar with and/or tracking some of these common KPIs:

  1. Monthly Active Users (MAU) and Daily Active Users (DAU)
  2. Customer Retention Rate (CRR)
  3. Net Promoter Score (NPS)
  4. Feature and Product Adoption
  5. Customer Satisfaction Score (CSAT)
  6. Customer Lifetime Value (LTV)
  7. Customer Acquisition Cost (CAC)
  8. Customer Conversion Rate

These metrics are powerful tools and absolutely essential to understanding and shaping your product. So what's the problem? Most of those metrics are either lagging indicators or reactive.

By the time enough product sales have accumulated and your department is asking for NPS, you're one to three months post launch. And there's very likely a big gap between the score you were aiming for and the one you're getting now. After all, there are a lot of factors that influence customer satisfaction outside of the product itself, like the purchase or support experience — things you'll be held accountable for but wouldn't have been able to account for.

You know that the success of your product and your career hinges on continuously deepening your knowledge of your product, your business, your industry, and your customers. But there's just so much you can't know. If that uncertainty keeps you up at night, you're in good company.

How do you know if people will like your product? How do you close the gap between what you think you know and how it all shakes out after you've launched? What role can metrics play in helping you find the answers to these questions proactively instead of waiting for star ratings, reviews, or support tickets to come through?

The good news is that there really is a simple way to get real-time insight about your product — one that helps you accurately predict your product's success and calls attention to issues you can fix before they impact satisfaction scores after launch.

The great news is that if you're user testing (and most product managers are), then you're already halfway there. The future of product management metrics is deepening the insights you get from user testing.

Let us show you how future-minded product managers are closing gaps in knowledge and enhancing their expertise of their customers, product, and industry through a whole new generation of user testing metrics.

Reenvisioning User Testing Metrics

"When the product succeeds, it's because everyone on the team did what they needed to do, but when the product fails, it's the product manager."
Marty Cagan, Inspired

This quote explains exactly why being a product manager is so stressful. You have a lot on your shoulders. It's not always fair, but it is part of the job.

User testing outcomes align with product management outcomes, which is why the majority of product managers are performing user tests on a semi-regular basis. It eases some of that pressure by granting peace of mind.

But if you're only treating your user tests as a gut check to surface glaring issues before launch, you're missing a powerful opportunity to unlock deep insight into your product and how successful it's going to be. By exploring your user testing metrics, you uncover some phenomenal benefits, including:

  • Forecasting product success
  • Ensuring you're catching the most critical issues
  • Tracking how customer input is influencing innovation and product discovery
  • Demonstrating how customer feedback is shaping your product

In short, you get the good stuff.

Here's where it gets a little tricky: a lot of product managers recognize that there's utility in user testing metrics, but 70% struggle to track them. This is either because they

  • don't have time to calculate them
  • don't have benchmarks or consistent, trackable processes
  • don't know which processes to measure, or 
  • some combination of all three.

It's understandable because…well, these metrics can be really difficult to deliver. First of all, how do you decide which metrics are important for you to measure? Bugs collected? Bugs fixed? Surveys completed? How many active testers there are? How engaged they are? You can waste a lot of time by not knowing which metrics are the most impactful to track.

Then there's the matter of actually collecting these results and data signals from all over the place. If you aren't using an all-in-one platform, you're saddled with the time-consuming task of pulling data from multiple spreadsheets, tester emails, and tools. You might also need to rely on your stakeholders in Engineering, Sales, and Product Success, which means you're beholden to their timelines and bandwidth before you can even start.

All this work and you haven't actually gotten to the part where you're calculating the impact of your efforts. If you're a time-crunched product manager (and let's face it, most are), you probably don't have a spare few hours each week to locate all the data, clean it up, and manually perform the calculations.

I thought you said it was simple!

Stick with us, we're almost there.

For this reason, product managers end up taking two paths when it comes to their user testing metrics:

  1. Nothing (and feeling a little uneasy about it), or
  2. Relying on "vanity" metrics, i.e., numbers that look good enough to placate stakeholders during a presentation but ultimately don't tell you anything about your product or your customers when taken on their own. Examples of this include the number of bugs found or number of surveys completed.

But there's a third option that blows the other two out of the water:

  1. Tracking meaningful and impactful user testing metrics that enhance your understanding of your products and customers. This enables you to build better business cases, align your product team and stakeholders, and efficiently problem solve on new and existing product experiences.

How are the product managers who perform option #3 getting it done? That's right, it's magic!

Just kidding. These PMs aren't wizards. They're also not more intelligent, more capable, or getting less sleep than any of their peers who are struggling to do it all. Instead, they're using a two-part solution.

The first is that they're focusing on three core user testing metrics. The second is they're reinforcing their processes with automation and relying on smart reporting to stay on the ball as they run user tests alongside their other responsibilities. OK, that second part is a little bit like magic — but it's attainable even if you're still waiting on that letter from Hogwarts.

This approach guarantees you're getting a unique perspective on your product and your customers that's more nuanced and granular than broader, whole-experience-view metrics like NPS or CSAT. By working with the medium-sized data you get during user testing, you're empowered to come up with focused solutions, whittle down the guesswork in building business cases, and easily prioritize fixes and ideas.

Meanwhile, automation makes product management more…manageable. No surprises there; it's likely that you're already using some level of automation to deliver the metrics you're currently tracking. But until recently, there haven't been tools for product managers looking to automate their user tests — let alone automate the metrics that increase their visibility and showcase their impact.

But before we talk about the automation, let's look at the metrics themselves. Here are the three core user testing KPIs that are leading the future of product management metrics — the questions they answer, the benefits of tracking them, and how they build on each other to get you closer to your product management goals.

Product Success Score

Is my product/feature/release going to succeed?

The Product Success Score is an estimated measure of confidence based on the feedback within the project. As you fix issues, implement ideas, and receive praise, your success score will increase. And the more engagement you have, the more accurately this score forecasts your product's success on the market.

The challenge with measurements that product managers use to try and evaluate product and feature acceptance, like star ratings and NPS, is that they cover the customer experience, rather than the product experience on its own. In short, they aren't designed to forecast the success of your product. 

This also makes these scores impossible to calculate accurately before launch. Try and use them to predict your product's success before your product has hit the market and you often end up with a totally different number a few months later.

What makes the Product Success Score vital as a product management metric is that it offers contextual, easy-to-act-on insight about your product and the way customers experience each feature or area. By weighing the impact of positive and negative feedback submitted by target market testers, you have a holistic product success score to work with that enables you to track shifts in satisfaction and product acceptance.

Calculating Product Success

The Product Success Score relies on an impact measurement for each piece of feedback submitted by user testers. This impact is calculated by weighing factors like how important the feature is to the overall product experience (priority), how often issues occur (popularity), and how severe those instances are (severity).

The Benefits

  • The Product Success Score gives you the visibility you need to forecast your product's success to leadership and your stakeholders. It tells you whether or not your product is on the right track with customers without waiting for reviews or support calls to roll in.
  • With medium-sized data, it's faster to identify, tweak, and resolve sources of friction at a granular level. This enables you to increase market acceptance and satisfaction of your product as a whole.
  • This score accurately represents how your customers feel about your product on its own, without any influence from other parts of the customer journey. This enables you to predict customer acceptance before you launch or release.
  • This score lets you drill down to see how much impact specific product experiences have on product acceptance overall, making it easier for you to prioritize improvements by impact.

Delta Health Score

How confident can I be in my test results?

Taking the temperature of how your user test is performing can be difficult. It's usually done through gut feeling or benchmarking against past experiences of running tests. Neither of these are particularly helpful, especially when you have limited bandwidth to deal with problems (like low engagement, for example) when something goes wrong.

The Delta Health Score measures the level of engagement within your user testing project, based on the number of active testers, tester engagement, and feedback quality. It uses industry benchmarks to set a threshold for how much feedback (positive or negative) you need about product areas or features in order to have a good understanding of customer acceptance. It then grades the amount of feedback you've gotten over the course of your test against that threshold.

This score makes it immediately obvious when your project needs your attention, like whether a part of your product needs more tester feedback, whether or not you need to extend a phase of testing, or if testers are being blocked from testing the product.

The Benefits

  • This metric helps you keep your finger on the pulse of your test. A clear sign of when things are running smoothly, it allows you to focus on other product management tasks with more confidence. It also signals when it's necessary to shift your efforts towards driving tester focus to a neglected product area.
  • Product areas with a strong health score become the concrete proof you need to make stakeholder-aligned, customer-focused product decisions with confidence.
  • This score serves as a benchmark you can apply across projects. This makes it easier to identify and implement improvements to your user testing processes.

Test Impact Score

Am I making an Impact?

It is notoriously difficult for product managers to show their value in quantitative terms. It's even harder when you have limited control over the factors that influence user satisfaction, but you're still accountable to the overall success of your product.

The Test Impact Score measures the impact your efforts to address feedback from user testing have had on your product. This metric weighs the Product Success Score from the beginning of your project with what that score is currently to show how the issues you're fixing and ideas you're implementing are impacting overall satisfaction and acceptance over time.

Rather than pointing to how many issues have come in versus how many issues got fixed (which is useful in its own right), the Test Impact Score takes it one step further by showcasing the impact of those implementations over time. Not all changes made to a product have an equal impact on the user's experience, after all.
This score not only highlights how much influence your efforts have on your product but how efficiently you're prioritizing what needs to be done — the hallmark of a great product manager.

The Benefits

  • This metric is a great indicator of how well you're directing your efforts and how efficiently you're prioritizing. Monitoring movement in your Test Impact Score gives you the foresight to pivot in real time, ensuring your team stays focused on the areas that yield the greatest benefit.
  • No matter how your product performs on the market, you have quantitative proof of the positive difference your efforts are making. When your product is crushing it on the market, this score allows you to map it back to the specific improvements you and your team have made.
  • Test Impact Score provides a historical view of acceptance, how it's changed over time, and which fixes and improvements had the greatest positive impact.

Automating Your Metrics

Now that you understand the three core user testing KPIs and how they serve you and your product, let's talk about the second part of the future of product management metrics: automation.

Automation matters because, while you could manually calculate the core KPIs you just learned about, it's just not realistic given the time constraints and limited bandwidth of a modern product manager.

Here's how automated user testing metrics are making a difference for future-minded product managers.

  • It helps you make decisions in real time. Your user tests are only as useful as the insights you gain from it. Automating your metrics lets you easily and instantly see this usefulness, like how many improvements you're making to your product. It also highlights how your test is running, so you know when to take action and when to relax. Dashboards and metrics give you the context and information you need to make data-driven decisions quickly.
  • It takes triage and calculation off your plate. User testing metrics are critical for understanding how your customers are shaping your product, but delivering those metrics is tedious and time-consuming. Centercode lets you centralize your data, while automation crunches the numbers in real time. This frees up your bandwidth, which you can then dedicate to your other product manager responsibilities.
  • It delivers user testing metrics consistently and accurately. Automation doesn't need to be trained and doesn't need to wrack its brain remembering how it landed on a number it calculated two months ago. It's reliable and consistent in what it's measuring and how it's measuring it, and this empowers you to set dependable benchmarks for tracking your progress and performance.

And it doesn't end at metrics: product managers are automating the whole of their user testing processes with tools like the Centercode Platform. This enables them to centralize their data and optimize their processes, increasing the efficiency, visibility, and ROI in addition to clearly and accurately measuring these returns.

Want to see these metrics in action?