Are you wondering when you should stick to using simple AB tests and when you should reach for a more advanced experimentation technique? Well then you are in the right place! In this article, we tell you everything you need to know to understand when to use simple AB tests and when to use a more complex experimentation technique.
We start out by discussing how simple AB tests are designed and analyzed. After that, we discuss some of the main advantages that simple AB tests have compared to other experimentation techniques. We also discuss some of the disadvantages of simple AB testing. Next, we discuss some situations where it is a good idea to reach for simple AB tests rather than other experimentation techniques. Finally, we discuss some situations where it is not a good idea to use simple AB tests.
Note. The scope of this article is not limited to AB tests that have only one control group and one experimental treatment group. It also includes straightforward split tests that have more than two treatment groups.
How to design simple AB tests
How do you design simple AB tests? The first thing to know about simple AB tests is that they are randomized at the subject level. That means that each subject in the sample is individually assigned to a specific treatment.
The first step to designing a simple AB test is understanding the different treatments you will have in your test. In general, you should have one control treatment that represents what would happen if no changes were applied to the experience and one or more experimental treatment groups that represent what would happen if you applied a specific change to the experience. Once you know how many treatment groups you have, you need to decide what proportion of the subjects will be allocated to each treatment group. It is common to use an even split here so that each treatment group gets the same number of subjects, but there are some use cases where you would want to use a different allocation.
Once you determine how subjects will be allocated across treatments, all you have to do is wait for subjects to come in and randomly allocate them to different treatment groups in the proportions that you predetermined.
How to analyze simple AB tests
How do you analyze simple AB tests? In general, simple AB tests are some of the most simple experiments to analyze. Just like AB tests are randomized at the subject level, they are also analyzed at the subject level. That means that each subject in your experiment contributes one observation towards your final results.
Simple AB tests are usually analyzed by using straightforward statistical tests to determine whether the observed difference in key outcome metrics is statistically significant. Here are some examples of statistical tests that can be used to analyze the results of simple AB tests.
- T-test. If your AB test only has two treatments and your outcome is numeric.
- Anova. If your AB test has more than two treatments and your outcome is numeric.
- Chi-square test. If your outcome variable is categorical.
Advantages and disadvantages simple AB tests
What are some of the main advantages and disadvantages of simple AB tests? In this section, we will discuss some of the main advantages and disadvantages that simple AB tests have compared to similar experimentation techniques.
Advantages of simple AB tests
What are some of the main advantages of simple AB tests? In this section, we will discuss some of the main advantages of simple AB tests.
- Simple to design. One of the main advantages of simple AB tests is their simplicity. They are generally straightforward to design, which means that you do not have to spend as much time and energy weighing complex tradeoffs in the design process.
- Straightforward to analyze. Just as simple AB tests are simple to design, they are also simple to analyze. There is not as much ambiguity in how the results of these experiments should be analyzed, so you also do not have to spend much time and effort weighing complex tradeoffs in the analysis phase.
- Broadly used and well understood. Another advantage of simple AB tests is that they are broadly used and generally pretty well understood. That means that it will be easy to find colleagues who can give you meaningful feedback on your AB test. That also means that stakeholders are more likely to understand how to design them and be able to self serve.
- Do not require an excessively large sample size. Another advantage of simple AB tests is that they do not require an excessively large sample size. This is especially true when you compare them to more advanced experimentation techniques like multivariate experiments.
- Not easily disrupted by anomalous events. Another advantage of simple AB tests is that they are not as largely impacted by anomalous events that happen over short periods of time as some other experimentation techniques. This is because subjects are randomly assigned to the control group and the experimental group(s) at the same proportions throughout the course of the experiment. That means that all groups in the experiment should be equally impacted by most anomalous events.
- Not complicated to run simultaneous experiments. Another advantage of simple AB tests is that it is not complicated to run simultaneous experiments on the same group of users. This is the case as long as there are no major incompatibilities between two experiments, such as one experiment testing a change to a module that another experiment intends to remove.
- Design can be fully specified ahead of time. Another advantage of AB tests is that the design can be fully specified ahead of time. That makes it easier to plan for other AB tests and changes that you are planning.
Disadvantages of simple AB tests
What are some of the main disadvantages of simple AB tests? Here are some of the main disadvantages that simple AB tests have compared to more advanced experimentation techniques.
- No information on interactions between different changes. One disadvantage of simple AB tests is that they do not provide information on interactions between different changes that might be applied to a single module or experience. If you test all of the different changes that you want to apply to a single experience individually, you will not know whether there are synergies between treatments across the different experiments. There may be cases where combining the treatments that perform best in individual experiments creates an experience that is less favorable than another experience that is made up of treatments that do not perform well individually, but do perform well in concert with each other.
- Cannot account for complex dependencies between subjects. Another disadvantage of simple AB tests is that they cannot account for situations where there are complex dependencies between subjects. If you are in a situation where the behaviors of subjects in your experimental treatment group(s) have the ability to affect the behaviors of subjects in your control group, then your control group will be polluted by the actions of the treatment group and your results will be biased.
When to use simple AB tests
When should you opt for simple AB testing over more advanced experimentation techniques? Here are some examples of situations where you should opt for simple AB tests.
- When you are running a straightforward experiment without major sources of bias. You should generally think of AB testing as the default option that you should reach for when you are running an experiment. That means that you should always default to running simple AB tests unless there is a particular reason that you need to reach for a more complex design. Simple AB tests are easy to design, run, and analyze, so they should be your go to tool for experimentation.
When not to use simple AB tests
When should you use a more advanced experimentation technique rather than using simple AB testing? Here are some examples of situations where you should reach for a more advanced experimentation technique.
- When there are complex dependencies between subjects. One example of a situation where you should avoid using simple AB tests is when there are complex dependencies between different subjects in your experiment and the actions of one subject have the ability to affect the actions of another subject. This often occurs when you are operating in a two-sided marketplace, when subjects are competing for a limited pool of resources, or when there are complex network effects at play. In these scenarios, you are better off using experimentation techniques that are designed for these situations like staggered experiments or switchback experiments.
- When you want to modify multiple facets of the same experience. Another example of a situation where you should avoid using simple AB tests is if you have multiple different facets of the same module or experience that you want to modify. For example, if you were running an experiment on the signup button that a subject clicks to sign up for a service or a product, you might want to test several facets of the button, such as the button color, the button size, the text on the button, and the position of the button on the page. If there are multiple facets of a module or experience you want to test simultaneously, you are generally better off using a multivariate experiment.
Related articles
Other experimentation techniques
- How to choose an experimental design
- When to use multivariate experiments
- When to use staggered experiments
- When to use switchback experiments
