Hiring? Here Are 7 A/B Testing Interview Questions To Ask Potential Candidates

Are you hiring? Regardless of whether you’re looking to backfill an existing role on your A/B testing team, or your team is in the midst of growth, the process of interviewing an A/B testing candidate can be just as exciting as it is time-consuming.

Based on our own experience hiring conversion rate optimization consultants, we’ve found that good A/B testing hires, particularly at bigger companies, must possess a mix of technical chops and soft skills like communication and people management, in order to be successful.

Here are a few of our top A/B testing interview questions to ask when evaluating potential new hires for your team.

Why do A/B testing? Why is it important?

Ah. So simple, and yet so insightful. This question ensures the candidate’s technical expertise in testing and optimization. It also gives you insight into their perspective on the philosophical role testing plays in an organization. For instance, in a data-driven organization, testing may play a critical role in new product development. In other less-data oriented organizations, testing’s impact may be limited to just optimizing website. Regardless, understanding a candidate’s answer to “why do A/B testing” is critical to assessing whether or not they will jive with your company culture.

Describe a losing test that you learned the most from.

Not every A/B test will result in a win. But we’ve found that knowing what doesn’t work can give you just as much information as knowing what works.

As you’re listening to their response to this question, take note of the depth of their answer. Does the candidate provide a detailed account where a losing test revealed hidden problems? Or perhaps they discovered that a “best practice” doesn’t actually resonate with the majority of their customers.

This question assesses whether a candidate is willing to learn from every test, and try to uncover why a certain experience won or lost.  

You just discovered a JavaScript error somewhere in your code. Walk me through your process of debugging it.

If you’re hiring a developer, this question will not only help you uncover a candidate’s technical abilities, but it will give you an insight into how they approach solving problems.  Additionally, because there isn’t a specific answer to this question, how they answer will help you understand whether their approach complements your team.

Can you provide an example of when testing led you to provide insights in addition to outcomes?

In statistical terms, an insight gives more of a plausible causation, while an outcome provides an obvious association. Outcomes are typically demonstrable while insights require a deeper level of thinking that, in turn, yield answers that are more comprehensive and therefore more helpful for you.

Being able to produce both insights and outcomes are essential skills for anyone analyzing A/B test results. These skills are indicative of a person’s analytical prowess and the lens with which they examine results.

Here’s one e-commerce example: In an A/B test of a checkout experience, you decide to test the effectiveness of displaying the total amount of savings for which a customer is eligible. The outcome of your test would be the increase in revenue from this new experience. The insight might be that the display of total savings in the cart led customers to attribute a higher value to items purchased, which made them more likely to purchase higher priced items in the future.

Describe the meaning of “alpha” in relation to running statistical hypothesis testing.

It is important for any candidate to be familiar with the specific terminology and technical procedures, but especially in A/B testing. Specifically, alpha, also known as the significance level, is key because it helps measure how likely your results are the result of chance. An incorrect application of this term could be harmful, particularly if you’re relying on A/B testing data to influence important company decisions. Other terms to make sure they’re familiar with include minimum detectable lift and confidence and fixed time horizon.

Describe the ideal strategist/analyst/or developer you’d like to work with.

Often when people are asked to describe a hypothetical individual, they will unconsciously identify characteristics within themselves. This will help you assess the person’s overall character, but also whether they have qualities that would complement your existing team.

Describe a time when you’ve helped a non-technical person with a technical problem, and how you ensured it made sense to the non-technical person.

Communication skills are essential in any workplace setting, but especially for A/B testing roles—which often require working directly with both technical and non-technical stakeholders. Communication skills are also surprisingly difficult to teach. If the role for which you’re hiring will be interfacing with a range of positions and teams, questions like these will make sure they are already equipped with effective communication skills.

Contrary to what most believe, interviewing can be just as stressful for the interviewer as for the interviewee. We hope these questions make the A/B testing interview process a little easier for you.

These seven questions are a few of our favorite ways to get a feel for how a candidate would impact an A/B testing team. However, by no means are they the only questions you should ask.

Do you have a good A/B testing interview question to add to list this? Let us know in the comments.