Field Testing

What is field testing?

Field testing is a product development process for collecting usage data from potential customers before launch. This test type focuses on the unguided, natural, contextual usage of a product. Field testing harvests ongoing qualitative and quantitative customer feedback in real environments over an extended period of time. By gathering long-term engagement data, you’re able to capture behavioral, attitudinal, and longitudinal trends related to feature adoption. This is the data you need to understand and maintain user perceptions over time, as well as analyze and hone your machine learning algorithms.

Why is field testing important?

Field testing is a great way to experiment with your product just before launch in what is typically a limited availability window of product development. By getting your product into the hands of your target market after the alpha and beta phases of product development teams, field testing allows you to:

Access natural, unguided feedback

During a field test, you’re taking a more observant role. Instead of following your directions, users interact with the product on their own. This allows you to observe their preferences and habits. What features do they like or ignore? Do they stay interested in the product after using it for a couple days or weeks?

Fuel AI and machine learning with natural-use data

When it comes to machine learning, field tests allow you to see how your product learns from and adapts to your customers’ individual needs. Modern machines are getting smarter and smarter, but people have to actually use them in order for them to learn and grow. Field testing allows your product to learn and interact with people at scale so you can gather analytics (and matching behaviors) to seed machine learning and enhance your product's capabilities prior to launch.

Test marketing messages and customer support systems

Many companies leverage limited-release or soft-launch periods like field testing to get feedback on live marketing materials and test out how customer support systems will work. By the time it's ready for soft launch, the product should be free of major issues, making it a perfect time to really get feedback on these systems before you start to get some volume or traffic from general release.

Who does field testing?

Field testing is often handled by product managers, support teams, and product marketing. These teams generally have the most access to the product at this later stage in development and have the most to gain from the insights gathered.

When does field testing happen?

Field testing occurs after you’ve completed successful alpha and beta tests. This phase is typically known as limited availability, early release, or soft launch. During this limited release phase, the product shouldn't encounter any major issues, so it's prime for testing in live or production environments.

How to run a field test

Like other forms of user testing, field tests follow a standard, multi-phase approach. These phases are: planning, recruitment, testing (engagement, feedback, results), and wrap-up. This structure allows teams to account for the pace of project activity and plan when to expect results.

Time is usually tight in the later phase of development, so maintaining a schedule is key. Your project manager will thank you for having a clear understanding of the different phases of your field test.

Planning phase: Every project starts with a plan. This is a simple and concise document that outlines the objectives or intent, schedule, and ideal tester profile. During the planning phase, gathering input from stakeholders from product management, software quality assurance, engineering, user experience, and project management is ideal as a jumping off point to discuss test plan details. Check out this free test planning kit.

Recruitment phase: During the recruitment phase, the team managing the field test will identify ideal candidates for testing and start reaching out to invite them to join the project. While the process can differ between organizations and by internal vs external testers, they will all likely need to go through some variation of the recruitment stages: application, tester selection, and tester onboarding. Check out this free recruitment kit.

Engagement management: Communicating expectations to testers is key for field testing, just like all other methods of customer testing. During the engagement phase of field tests, messaging or resources are available to testers communicating what is expected, guidelines for product usage, and what needs to be completed by a tester to be considered compliant (i.e., actually participating).

Feedback management: As engagement picks up and testers start interacting with the product, feedback will begin to flow in. Engaged testers will provide ample product feedback. But in order to make the most out of that feedback, it needs to be triaged and cleaned up. This means: organizing it by product area, ensuring it has enough actionable detail, and prioritizing it based on frequency and severity. This ensures your stakeholders can understand what needs to happen and start improving your product as early as possible. Here are some tips for handling tester feedback.

Results distribution: Results should be provided to stakeholders on a consistent basis. Daily and weekly are the most common types of distributions for data. The results from testing typically include: tester feedback overviews, surveys, and tester engagement outcomes. It's best to work in one to two week sprints in order to allow time to complete tasks and a reasonable time to provide results to your team members.

Wrap-up: The final phase of field testing is the wrap-up period, when you tie up any loose ends and bring the test to a close. During the wrap-up phase, teams will need to announce the closure to testers, distribute incentives (if applicable), retrieve products from testers (if applicable), and distribute to comprehensive results from the project. Here are some resources to help with the closure of your test.