Launched Your A/B Test? Great! Here’s What’s Next

Launching an A/B test is not a matter of hitting go, and then putting your feet up while the data rolls in. Here are some things to incorporate to enhance your post-test launch process.

Whether you’re a seasoned experimenter or an entrepreneurial tester, launching an A/B test or any type of experiment is a major milestone in your testing cycle. After all those weeks of planning, strategizing, conceptualizing, developing, and QA-ing, you might feel ready to kick-back, relax, and watch your data accumulate as the test does its thing.

That might sound great, but there’s actually more that you should do after launching that can really enhance your experimentation program and take your testing to the next level.

Step 1: Post-Launch Validation

Many organizations deploy an A/B test QA process prior to launch but the post-launch QA is just as crucial. Why? There’s almost always a time gap between pre-launch QA and actual launch. That means there’s almost always a risk of a site update that could affect the test code. A post-launch QA process is a critical step to ensuring what you’ve launched is running as expected.

There are two moments post-launch where validation is critical.

The first is during the soft launch. With any test launch, it’s always a good idea to do a “soft-launch” where you only release the test to a certain portion of your website traffic. At Brooks Bell, we typically launch tests at 25% of the targeted level, meaning that if you were to release the test to 100% of your users, only a quarter of those users would actually receive the test experience. This reduces any risks of bugs (or even worse–a broken site experience!)

Metrics are the second thing you’ll want to monitor and validate during your soft-launch. In any test setup, it’s very common to set custom metrics to keep track of any new or existing elements that aren’t tagged on your site. During the soft launch, your analyst should check to make sure those metrics are firing as expected. This ensures that, come report time, they won’t be put in the embarrassing position to tell your team that a metric wasn’t tracked correctly.

Recommended Download: Improve Your A/B Test Quality With Our QA Checklist

Step 2: Record & Communicate

During post-launch QA, it’s best practice to take screenshots of the control and the challenger experiences. If any issue arises during your testing period, you’ll have a record that the test code was clean when it launched, and therefore, you can point to a site update being the culprit behind the issue.  

Another benefit of taking these screenshots – you now have images to communicate to your stakeholders that your test has launched. In any experimentation program, building momentum and awareness is key to success, and communication about a successful test launch is just one part of building trust with your organization.

Whether it be an email or a newsletter article or just a post within Slack, providing a brief description of the test along with the hypothesis and the screenshots will go a long way.

Step 3: Plan Ahead

As much as we’d like to sit back and relax after a test launch, many testing leaders simply don’t have that kind of luxury due to the high demand placed on experimentation programs these days.

In any type of experiment, experiment duration is always a limiting factor when it comes to helping you achieve your program goals. To make the most out of this time, you should use the next two weeks to update your testing roadmap with the latest timelines, plan out when you need to deliver the report for the test you just launched, and consider implications of a test winner and what your team should do about it. 

Step 4: Celebrate!

Let’s not forget that launching a test is an accomplishment. It takes many hours of strategizing, planning, designing, and coding to make a test happen. And most likely, you didn’t do it all by yourself.

At Brooks Bell, we acknowledge a test launch by sounding the gong in our office and announcing on Slack for those who aren’t in the office.

In any organization, creating a strong testing culture fuels growth and as mentioned before, momentum and awareness is your key to success. Most organizations set their experimentation program goals based on how many tests they’ve launched in a year. So don’t be shy! Celebrate not only your wins but your test launches as well!