The One A/B Testing Tool You Need To Run A More Impactful Optimization Program

Over 15 years in the experimentation space, we’ve worked with many testing program leaders. We’ve observed their strategic approach, their dynamic with other teams, and the behaviors they embrace to guide their success.  We’ve found that there’s one practice common to all the most successful, influential leaders.

It may surprise you, but the secret weapon to running a high-impact optimization program is documenting their A/B tests within a central knowledge base.

Why a Knowledge Base? 

There are many benefits of establishing an A/B testing knowledge base, from everyday reporting to sharing powerful insights that can impact and influence the broader organization.

Establish a Single Source of Truth
Rather than hosting information in a million different places, a knowledge base makes your learnings more accessible for other teams and leaders to make data-informed decisions.

Ease Onboarding
Historical A/B test data is critical to getting your new team members up to speed on what tests you’ve already run and any insights you might have already learned about your customers.

Prove Testing ROI
Having a system-of-record for your A/B tests enables you to communicate your program’s impact, arming you with rock-solid data to help you defend and even increase your budget. In the same vein, this can also help you show the value you’ve created as the testing program lead and thus providing you with the data you need to land that big promotion.

What Data To Include in Your A/B Test Knowledge Base

Now that you know why putting your A/B tests in a knowledge base is important, let’s talk about how to do it.

Every experiment record has a mix of super technical data and high-level summaries. Deciding which information to include in your A/B test knowledge base ultimately depends on who will be using it and how you want the information to be used.

Working with our clients, we’ve seen many different approaches to establishing a central source for A/B test results. Ultimately, we’ve found it’s most important that you strike a balance between big-picture summaries and hard data.

Here’s what fields you should include in each A/B test record:

• The launch and end date
• The page(s) being tested
• The primary and secondary metrics
• The problem or opportunity that inspired the test idea
• How the variation(s) differed from the control
• Visuals of the control and variation(s)
• Performance data, initial outcomes and associated confidence levels
• What insights you learned about your customer
• Next steps

Selecting A Tool To Store Your A/B Test Results

From Confluence and Trello to spreadsheets and slide decks, we’ve seen it all when it comes to how optimization team leads have jerry-rigged solutions for storing their A/B test records.

Oftentimes, establishing a new system at any company is to look for solutions within your existing toolsets. This is what’s great about using solutions like Google Drive, SharePoint and Confluence. They’re often deeply embedded in your organization, everyone has access, and they come at no additional expense.

Take this Reddit user, for instance:

“Best practice (IMO) is to standardize and set up breadcrumbs [within a spreadsheet]. Make sure the raw event data never gets wiped, store the analyses files consistently (got repo of jupyter notebooks, named BI dashes, folder of spreadsheets, whatever), do a 1-page write up that gets sent around with the TL;DR at top and links to the analyses at bottom, and then throw links to everything with the TL;DR in a master spreadsheet. ”

Or this one:

“We‘re currently not running too many tests, which is why a simple spreadsheet works. Since I‘m planning to do more [tests], I‘m looking into solutions that would scale better. Currently leaning towards a tool like Notion or Airtable, where I can build a database of tests and include relevant information (test description, metrics, tradeoff metrics, runtime, sample size, etc) but also visualize it differently, for example, in the form of a kanban board.”

On the other hand, these patchwork solutions always feel a little…hacked.  There will always be a handful of tasks that require a lot of time and manual work on your part in order to make it work.

Recommended: Quickly Create an A/B Test Knowledge Base Using This Simple Excel Template

Regardless of which combination of tools you decide to use, it’s important to set up your knowledge base with these qualities in mind:

It should be accessible. Experimentation results and insights should be easy to access, regardless of role or level in your organization.

It should be intuitive. If you want to encourage people to interact with your tool, it needs to be intuitive, with user-friendly features like filters, search, high-level summaries and helpful visuals.

After all, if you’re the only one who can access and use the knowledge base, then it really isn’t a “centralized knowledge base” after all.

Finally, it should support iteration. Your knowledge base should be able to link previous learnings to future tests. The best way to gauge this is to go through the process.

Review your last A/B test record and ask yourself:

• What did you learn?
• What new questions did you uncover?
• Did this test uncover an intuitive iteration to expand upon?
• Is all of this clearly communicated? Would your bosses’ boss understand the value?

Looking for a solution that doesn’t require a ton of manual work?

Check out Illuminate™, our proprietary software for storing, sharing and learning from your A/B tests.

Recommended: 4 Challenges To Building & Scaling Your Testing Program, Solved With Illuminate

We built Illuminate specifically with testing program leaders in mind. It’s intuitive UI and software integrations make it easy to set up experiment records, build slick-looking (and executive-friendly) A/B testing case studies, keep track of your customer insights, monitor overall program metrics and—most importantly—share all of this information across your entire organization.

Ultimately, the secret to getting ahead is getting started.

If you’re looking to build an A/B testing knowledge base, then the best advice we can give is to do it—right now. Don’t over-engineer it. Don’t get too caught up in the details. In fact, we suggest starting with this basic Excel template. Once you’ve codified the process a bit and identified areas of improvement, you can evolve it from there.