Can You Trust The Results of Your Website Tests During a Pandemic?

Over the last few months, gathering with testing and optimization leaders in weekly, virtual roundtables to discuss the various challenges we’re facing. Week over week, in nearly every single conversation, the same question has been raised: Can you trust your website test results during this time? The answer is, of course, complicated, but that doesn’t mean that one doesn’t exist.

Throughout the pandemic and resulting economic fallout, testing teams have proven critical to businesses. This is thanks, in part, to their ability to quickly test and launch different types of messaging, calls to action, and– through testing-influence the rapidly changing directions of their companies. 

And yet, from the early days of the COVID-19 outbreak up to today, even as some parts of the world begin the process of re-opening, questions remain as to how much you can trust your test results, and whether or not these results will be able to withstand the test of time. Here’s our take on the issue.

Download the main takeaways from the last eight weeks of our virtual roundtable conversations

First, assuming you’ve done all the work of identifying a variable to test, creating variants(s), splitting your traffic randomly, determining your sample size based on a set confidence level and MDL and all that jazz, then yes, you can say your results are both valid and reliable. 

Running a test during a pandemic, economic downturn or any other type of crisis doesn’t negate the results. It only raises questions as to what insights you can extrapolate from the results. 

So then, the real question you should be asking is this: in “normal” circumstances and with typical audience breakdowns and traffic levels, would your winning experience actually win? Would that flat test actually be flat?

To understand this, you need to define “normal” for your business in regards to your website and understand the full picture of how that has changed over the past few months.

Take a look at your website metrics.

How has your audience changed in the last eight weeks? How has their behavior(s) changed?

A recent article by Bain & Company concluded that “nuanced customer segmentation has given way to the tough reality that right now, for many businesses, there are just two key segments: unemployed and working from home.”

That might be a bit of an oversimplification, but there’s no denying that the pandemic has greatly changed customers’ needs and may have also introduced new customer segments to your website.

To get a full picture of these changes, do an in-depth analysis of visitor and customer profiles and behaviors. Specifically, look for any changes in behavior, such as:

• Referring channels
• Organic keyword referrals
• Demographics
• Location(s)
• Product categories
• Cross-category browsing
• AOV
• Time on page
• Checkout flow engagement

For new customers, there are many environmental and market situations that may lead to different behavior than your typical customers. Their long-term and short-term needs might also be different and can inform how you track those needs.

Regardless of if your company is seeing an increase or decrease in demand at this time, you should consider pairing this analysis with qualitative user research to build a COVID Customer Profile.  This can serve as a guiding light for your optimization strategy during this time, and can also help you define and track analytical indicators that would suggest a return to “normal” customer behavior. 

We’re of the opinion that if you’re not testing, you should be. If the above analysis turns your existing test strategy and roadmap completely on its head, then here are a few guidelines to get you back on track.

1. Be intentional about what you want to learn about your customers during this time. 

Looking at your testing roadmap, consider what represents an opportunity for optimization versus a true test-and-learn opportunity?

Eliminating friction in the checkout process should be a top priority, of course. Or, perhaps, consider testing the addition of elements that build confidence and help customers complete the funnel.

This may also be a good time to consider reprioritizing all the UX tests that were previously at the bottom of your testing queue.

2. If you’re seeing a significant increase in traffic, use your Holiday Testing Playbook as a guide.

Though the environment is different, there are clear parallels between the two time periods: short windows, visitors with different motivations, and different behaviors.

As always, your goal should be to get as much value out of your traffic as possible, but treat any insights as you would treat any insight gleaned from testing during the holidays. 

Download: 5 Testing Tips for the Holidays

3. Plan for re-testing or use holdout groups to measure the effect of new feature launches.

You’ll want to re-test certain types of experiments (like pricing tests) that could be influenced by users’ mindsets, particularly as they relate to the economic outlook.

In the case of launching new features, we’ve heard in these roundtables that many companies are simply launching and then planning to do a pre- and post-launch analysis.

However, an alternative route would be to launch to 95% of users and use a 5% holdout group to measure performance over time. Just remember that your holdout group cannot be subject to any more website changes in order to get a true understanding of impact. 

4. Take a long term view of the opportunities created by the situation at hand.

Earlier on, the pandemic presented an opportunity to run tests in a unique environment and find out what works during an economic downturn. 

Now, because we really don’t know how long this whole thing could last (and whether states “re-opening” will lead to a surge in infections and force us to retreat inside again) you may want to consider the long-term effects of the pandemic as a whole.  Will it create a new mindset for future generations and resonate with them the way the Great Depression resonated with those who lived through it?

As we navigate this new normal, you can stay at the forefront of audience segmentation and personalization by considering the long-term effects of the pandemic as a whole, and whether it will create a new mindset for future generations. The insights that you gather today can help shape your user-profiles and personas in the future.

All in all, when it comes to running tests during this strange, sad time, our ultimate advice is to keep moving ahead with cautious optimism.

If a test has already started, keep it running. Evaluate others on a case by case basis, plan for multiple versions of the future.

And finally, remember that, to some degree, there is always a timestamp on your test results. You have to get into a cycle of constant, iterative improvement, which sometimes means testing and re-testing experiments.

If you’re ready to start planning for recovery, we’re here to help.

Fill out the form below to request a free consultation to discuss how your customers and their expectations have changed, how to think about segmentation and personalization in the wake of COVID-19, and how you can capitalize on a market recovery.