Are you wondering what CUPED stands for in the context of experimental design? Or maybe you are interested in learning about when you can and cannot apply CUPED in an experiment? Well either way, you are in the right place! In this article, we tell you everything you need to know to understand when to apply CUPED in an experiment.
We start out discussing what CUPED is and when CUPED can be used in an experiment. After that, we discuss some of the advantages and disadvantages of using CUPED in an experiment. Next, we provide some examples of situations where it is a good idea to use CUPED. Finally, we provide examples of situations where you should not use CUPED.
What is CUPED?
What is CUPED? And how do you design an experiment using CUPED? We will start out by discussing what CUPED stands for in the context of experimental design. The acronym CUPED stands for “controlled experiment using pre-experiment data”. The general idea behind CUPED is that when you analyze your experiment, you can supplement the data that you collected during your experiment with additional data that was collected before the start of the experiment. In doing so, you can reduce the variance of the coefficient estimates in your experiment and effectively reduce the sample size that is required to reach statistical significance for a given effect size.
The general intuition behind how CUPED works is as follows. In your experiment, you will have one primary outcome variable. The primary quantity you are trying to estimate in your experiment is the difference between the value of that outcome variable in the control group and the treatment group. The larger the variance of that quantity is, the more samples you will need to ensure that you have an accurate estimate of that difference.
Now imagine that the variance of your estimate can be decomposed into two pieces, one portion of the variance that is due to random fluctuation in your primary outcome and another portion of the variance that is due to the effects of your treatment. The idea behind CUPED is that you can use pre-experiment data to extract information on the first portion of the variance that is due to random fluctuation in your primary outcome. This allows us to explain some of the variance without looking at post-experiment data and effectively reduces the number of post-experiment data points needed to estimate the variance of our quantity of interest.
Note that the pre-experiment data should be collected on the same sample of subjects as the post-treatment data. That means that you may not be able to apply this method if you are not able to collect pre-experiment data on the subjects in your experiment.
How to design experiments with CUPED
CUPED is a post-treatment variance reduction technique that can be applied after treatments have already been applied. You do not have to make many modifications to the actual design of an experiment and the strategy that is used to randomize subjects into treatment groups. That being said, there is one key design decision that needs to be made in order to use CUPED for variance reduction. That decision revolves around what pre-experiment data you will include in your analysis in order to achieve variance reduction.
So how do you decide which pre-experiment data to include in your analysis? In general, you want to include pre-experiment data on a variable that is not impacted by the treatment but is highly correlated with the primary outcome of the experiment. The more highly correlated the control variable is with the primary outcome, the larger the reduction in variance will be. It is common practice to use pre-experiment data on the primary outcome of the experiment as the control variable because it is as highly correlated with the experiment outcome as you are going to get. The pre-experiment data on the primary outcome is not impacted by the treatment because the treatment has not yet been applied at the time it is collected.
Another important design decision that needs to be made is what window of time will be used to capture pre-experiment data on your subjects. The exact details of the window that should be used will vary based on the application area and the domain within which CUPED is applied. In general, longer windows give you a better chance of observing meaningful outcomes in the pre-experiment window. That being said, larger windows also increase computational costs and potentially reduce the pool of subjects available with data during the full pre-experiment period (especially if many of your subjects are new to your product).
How to analyze experiments with CUPED
Experiments that use CUPED for variance reduction are not as straightforward to analyze as simple AB tests that are run without any variance reduction. That being said, CUPED is a common technique that is employed in many companies across multiple industries. That means that there are plenty of resources available to help you understand how to analyze experiments that use CUPED. In general, you just need to use a slightly different formula to calculate mean and variance estimates for your analysis.
Advantages and disadvantages of using CUPED
What are some of the main advantages and disadvantages of using CUPED for variance reduction? In this section, we will describe some of the main advantages and disadvantages of using CUPED.
Advantages of using CUPED
What are some of the main advantages of using CUPED in a randomized experiment? Here are some of the main advantages of using CUPED.
- Run experiments with smaller sample sizes. The main advantage of using CUPED to analyze the results of a randomized experiment is that you can run your experiment with a smaller sample size and still be able to detect changes with the same effect size. This is a large advantage that can bestow great benefits upon the experimenters because it allows them to make good decisions faster.
- Does not impact the randomization scheme. Another advantage of CUPED is that it is a post-treatment method that can be applied after subjects have already been randomized into treatment groups, even in situations where you did not plan to use CUPED ahead of time. It does not impact key elements of the experimental design such as the randomization scheme that is used in the experiment. This can be a huge advantage if your company has already built (or purchased) a solution to implement randomization in experiments, especially if this system is difficult to modify.
- One of the most well known variance reduction techniques. Another advantage of CUPED is that it is one of the most commonly used variance reduction techniques for randomized experiments. That means that it is not as difficult to find collaborators who can give you meaningful feedback on your analysis. It also means that stakeholders who are skeptical of technologies they have not heard of are less likely to provide pushback.
- Design can be fully specified ahead of time. Just like simple AB tests, designs for experiments that use CUPED can be fully specified ahead of time. For example, it is possible to calculate how long an experiment will need to be online ahead of time. This makes it easier to plan out your launches.
- Not affected by anomalous events. Since CUPED uses a similar randomization scheme to a typical AB test with a proper control group, it is not as strongly impacted by the effects of anomalous events as other experimental techniques. This is because any anomalous events that are observed should equally affect the control group and the treatment group.
Disadvantages of using CUPED
What are some of the main advantages of using CUPED in your experiment? Here are some of the main disadvantages of using CUPED in your experiment.
- Not as widely understood as simple AB tests. One disadvantage of using CUPED is that while it is one of the more commonly studied variance reduction techniques, it is not as well studied and commonly understood as simple AB tests. This means that there will not be as many people who can give you feedback on your analysis plans.
- Requires pre-experiment data on subjects. One of the main disadvantages of using CUPED is that it requires you to collect pre-experiment data on your subjects. That means that it may not be possible to use in situations where you do not have the ability to collect pre-experiment data.
- Additional design and analysis decisions need to be made. Another disadvantage of using CUPED is that there are some additional design decisions that need to be made before you analyze your data. This means that there may be additional time that needs to be put into validating assumptions and planning your experiment.
- Can make it more complicated to run simultaneous experiments. Another disadvantage of using CUPED to analyze your experiments is that it may make it more complicated to run simultaneous experiments. It effectively extends the time period over which you need to collect data for your experiment. If you are running other experiments that you suspect may have interactions with your current experiment and you want to be sure that the two experiments do not interfere with one another, you may have to consider whether to include a larger delay between experiments.
When to use CUPED in experiments
What are some examples of situations where it is a good idea to use CUPED in your experimental analysis? Here are some examples of situations where you should use CUPED in your experiments.
- When you have a small sample size. One example of a situation where it is a good idea to use CUPED in your experiment is when you have a small sample size and you do not have any way to increase the sample size. CUPED is a method that can reduce the amount of sample size that you need in your experiment, so it is well suited to these situations.
- You are expecting a small effect size. Similarly, it is also a good idea to use CUPED when you are in a situation where your expected effect size is small. The smaller your effect size is, the larger the sample size you need in order to be able to detect that effect. Methods like CUPED can help to offset the effects of having a small effect size and reduce the sample size that is required to detect a small effect.
- When you need to speed up experimentation. Another example of a situation where it is a good idea to use CUPED in your experiment is when you need to speed up experimentation and make decisions faster. If you reduce the sample size that is required for your experiment, then you reduce the amount of time you need to run your experiment in order to achieve the required sample size.
When not to use CUPED in experiments
When is it not a good idea to use CUPED to analyze your experiments? Here are some of the main situations where it does not make sense to use CUPED to analyze your experiments.
- When you cannot collect pre-experiment data. The main situation where it does not make sense to use CUPED in your experiment analysis is when you are not able to collect pre-experiment data on most or all of your subjects. If you are not able to collect pre-experiment data on most of your subjects then you will not experience many benefits from using CUPED. For example, if you are optimizing a website and are running an experiment that only affects new users, then it may not make sense to use CUPED because you will not have any pre-experiment data on new users.
- When you have a very large pool of subjects available. Another example of a situation where it does not make sense to use CUPED is when you have a very large pool of subjects available to run experiments on. If you are in a situation where you have a very large pool of subjects and you can already complete your experiments very quickly with that pool of subjects, then you do not stand to benefit as much from using variance reduction techniques. In these situations, you are often better off using simpler methods that are more commonly understood.
Related articles
Other experimentation techniques
Other articles on CUPED
- Examples of CUPED at Netflix
- Examples of CUPED at Booking.com
- Examples of CUPED at Uber
- Examples of CUPED at TripAdvisor
- Examples of CUPED at DoorDash
