A/B Test Code Quality vs. Quantity – Five Factors to Consider

Recently, someone gave me a zen drawing board. You use a brush dipped in water to paint on the board. After a few moments, the board begins to dry and your art fades. It’s an exercise in appreciating things for what they are in the moment. Finishing one piece of art, then moving on to the next.

As a Brooks Bell engineer who develops client-side A/B tests, sometimes my day-to-day work can feel similar. I build a user experience that runs on a website for a short amount of time and then is turned off and, in a sense, disappears.

Of course, that doesn’t mean developers writing A/B tests shouldn’t put effort into writing good code. But it does beg the question – does code for an A/B test need to be as well-written as more permanent features of your site? How thoroughly does it need to be reviewed and QA-ed? And, when there are many tests to run and limited time, what’s the happy medium between quantity and quality? Here are five factors to keep in mind when considering these questions.

1. Resources

Every optimization program has its limitations. Perhaps only one developer is on the team, but your test queue is long and your site traffic would enable you to run more tests simultaneously. Your developer isn’t able to keep up with the demand. Besides calling Brooks Bell for some extra development support, you might consider:

• Talking with your developer about how long certain elements of a test take to build. If things can be augmented or excluded without changing the overall strategy of the test, it may be worth simplifying in order to decrease development time
  
• Are there aspects of the development or QA process that could be simplified or expedited? With less dev/QA time, the likelihood of bugs with the test does tend to increase. But running more tests may also increase your chances of getting a big win, which may make this risk worthwhile. It’s a balance worth considering and discussing with your team.

Recommended Read: Five Telltale Signs of A Broken A/B Test

2. Timing

Some A/B tests are time sensitive. You may be testing something that is only relevant during a certain time of year, such as back-to-school promotions. Perhaps something happened in the news that is relevant to your business, and you want to quickly build a test relevant to that story. Or maybe during a certain part of the year, your website gets more traffic and you’d like to take advantage of that increase in visitors by running more tests. Some considerations along those lines:

• How can you plan in advance? Tests can be built ahead of time, then, shortly before you’d like to launch, have the developer/QA ensure everything is still functioning as expected.
  
• For last-minute requests, you’ll need to weigh risk vs. reward. Rushing development and QA could lead to issues with the test. But, if the test needs to be up as soon as possible, it may be worth launching and continuing QA while the test runs, fixing bugs as they are found.

3. Data Accuracy

As hard as we try to solve all the bugs during development and QA, sometimes there is still an issue with a test. Maybe something on the site was updated that wasn’t taken into account that is conflicting with the test’s code. Maybe a use case wasn’t recognized. Make there’s an issue in a certain browser. This isn’t ideal – nobody wants bugs on their site. But bugs can also affect your A/B test results.

Say a bug is found after a test is halfway done. The developer quickly fixes it. The test runs for the second half – and comes back flat – however, it seemed to be trending upwards for the second half. Now you’re in a tricky position – do you re-run the test with the fixed code, for the full duration, to see if you can reach significance with a confident win? Or do you just assume it’s a flat test?

It may have only been one small bug, but suddenly the results of the test feel inconclusive – and you’re not sure about the next steps. That wouldn’t have happened if the bug hadn’t been there in the first place.

4. Iterations and Reusability

Developers don’t like writing rushed code. We understand its necessity in some cases but rushed code tends to be messy code, and messy code is difficult to decipher. If your team tends to re-run similar tests often, tweaking just one or two things – then you’re better off ensuring your developer writes the cleanest code possible from the get-go.

Allowing the extra time upfront will save time in the future when the code needs to be updated – and your developer is able to more easily make the changes you need

5. Plans for implementation of winners

You develop a test, and it wins. Hooray! Depending on goals and resources, this may mean your team will:

A. Acknowledge the insight, but keep the test off for now with no plans to implement

B. Keep the test code running through the test tool, to capitalize on revenue gains immediately

C. Turn the test off and use the A/B test code to build the experience into the site

D. Build the experience into the site, but write new code with the goal of it functioning long-term

With options B & C it’s more important that the code quality of the A/B test is very high, since that code will be on your site long term.

With options A & D, the A/B test code is not a long term solution – it only needs to work for the duration of the test, so future changes, like elements that will be added to the page in the future, do not need to be considered in the same way.

There are many considerations to take into account when evaluating the quality vs quantity argument for A/B test code. It’s important to keep in mind the amount of risk you’re willing to take on – the risk of bugs in the code, risk of data inaccuracies – compared with what you have to gain – whether that’s getting more tests out, the potential for more wins, or just getting something launched within a tight deadline. Also, you should always be aware of the plans for any code written for A/B tests – will it potentially be iterated on, will winners need to function long term? 

Unlike my zen drawing board, the A/B test code may not be as temporary as you might think!.