What it is: The impact on an experiment or test that's caused by a researcher's expectations of the exercise. This can lead an experimenter to make incorrect conclusions about the test that are in favor of their hypothesis, despite the data or results.

Also known as: Experimenter-Expectancy Effect, Expectancy Bias, Observer Effect, Experimenter Effect.

What it isn't: Congruence bias. Although it's similar, the Observer-Expectancy Effect impacts more than just what we test, it can change the way we interpret data or subconsciously skew the results.

Why it matters: Being aware of this effect can help us as A/B testing experts to keep ourselves, our hypotheses and our preferred outcomes out of the picture—this will lead to cleaner tests, better learnings and, eventually, better KPIs.