Are you wondering how to set up a search engine optimization (SEO) experiment to understand how a change to your website will affect your search engine rankings? Or maybe you are interested in learning how to analyze data that has been collected in a SEO experiment? Well either way, you are in the right place!
In this article, we tell you everything you need to know about how to set up and analyze SEO experiments. We start out by describing what SEO experiments are and why they are important. After that, we talk about why SEO experiments are different from traditional AB tests. Finally, we discuss how to structure SEO experiments and how to analyze data from SEO experiments.
What is a SEO experiment?
What is an SEO experiment? A SEO experiment, or a search engine optimization experiment, is an experiment that is run to understand how a change to a website will impact search engine rankings. SEO experiments can be run to determine whether the search engine rankings of your website will increase if you apply a specific change to all of the URLS (or a specific subset of URLs).
For example, say you had a hypothesis that you could increase the search engine rankings of a website by shortening the titles on all of your URLs. You might run a SEO experiment to verify whether this is true before applying that change to all of the URLs on your website.
Why should you run SEO experiments?
Why should you run an SEO experiment before applying a change to your website? Even if common wisdom says that a change will have a positive impact on your search engine rankings, it is still important to verify that hypothesis with real data. There are a few reasons for this.
The most important reason to run SEO experiments is to avoid unknowingly introducing a change that negatively affects your search engine rankings. Search engine rankings can take a while to adjust and bounce back, so introducing a change that negatively impacts your search engine rankings to your whole website can be catastrophic. It is much safer to test the change on a smaller subset of URLs before releasing it to your full website.
Even if the change is not detrimental to your search engine rankings, it is still possible that the change will have no effect on your search engine rankings. If you are able to identify this ahead of time by testing the change out on a small subset of URLs then you can save yourself the effort of having to apply a large change to all of the URLs on your website. This will give you time to focus on other more impactful initiatives.
What makes SEO experiments different from AB tests?
What makes SEO experiments different from standard AB tests? Here are a few reasons that SEO experiments are a little more complicated than standard AB tests.
- Only one version of a page can exist at a time. The main reason that SEO experiments are more difficult to implement than standard AB tests is that a search engine will only consider one version of a page at a time. In order to run a standard AB test on a web page, you would randomly divide your traffic into multiple groups then show different groups of traffic different versions of the web page. You cannot do this with SEO experiments because you can only have one version of each page involved in the test. This means that you have to structure your tests in a different way that is more prone to bias.
- There is no fixed timeline when you can expect to see results. Another thing that makes it more difficult to analyze the results of a SEO test is the fact that there is no clear timeline for when you can expect to see changes to your search engine rankings. The amount of time that elapses between when you apply a change to your website and when you can expect to see a shift in search engine rankings can vary depending on the size of your website, the type of content you publish, the type of change you make, and many other factors. This makes it difficult to plan when you will analyze the results of your experiment.
How to structure SEO experiments
Do you use control and treatment groups in SEO tests?
So how do you structure an experiment to determine the impact that a change has on search engine rankings? Just like when you run a standard AB test, you will need to have a control group that shows what happens when you do not apply any changes to your website and a treatment group that shows what happens when you do apply changes to your website. This way you can determine whether any changes that you see in your treatment group happened because you applied the treatment or whether those changes would have happened anyways.
The main thing to keep in mind when determining how to structure a SEO test is that search engines will only index one version of a web page at a time, so you can only have one version of each page involved in your experiment. This means that you cannot randomly distribute your visitors into a control group and a challenge group. Instead, you have to randomly distribute the set of URLs you have to work with into a control group and a treatment group. For the URLs in the treatment group, no change will be applied. For the URLs in the treatment group, the change will be applied for all traffic.
How to determine which URLs are in the treatment group
How do you determine which URLs should be in your treatment group and which URLs should be in your control group when you are running an AB test? The answer to this question will vary depending on the number of URLs you have to work with.
- Large sample sizes. If you are working with a large pool of URLs that have similar characteristics then the path forward is relatively straightforward. You can randomly allocate the URLs into one group or the other without paying attention to the characteristics of any particular URL. If the pool of URLs you are working with is sufficiently large, then it is likely that the distribution of important characteristics (such as baseline traffic levels) will be similar in the control group and the treatment group.
- Small sample sizes. If you are working with a smaller group of URLs, then it is less likely that the distribution of important characteristics will be similar if you randomize the groups without paying attention to these characteristics. If you are in this scenario, then you may need to use a technique like propensity score matching to ensure that the distribution of important characteristics is similar in the control group and the treatment group.
Timelines for running SEO experiments
When you run an SEO experiment, there are multiple different time windows where you need to collect data from. Here are the main time periods where you should collect data.
- Before the treatment is applied (pre period, baseline period). The first time period where you need to collect data is the pre period, or the period before any treatment is applied to any of your URLs. The data that is collected in this time period serves as a baseline against which you can compare the data that is collected later on.
- After the treatment is applied (post period). The second time period where you need to collect data for an SEO test is the post period, or the period of time after the treatment is applied. You can compare the data that is collected during this time period to the data that is collected before the treatment is applied to estimate the effect that the treatment had.
How to analyze data from SEO experiments
What outcomes to use to analyze SEO experiments
What outcomes should you look at to understand the results of an SEO experiment? You have multiple options when it comes to analyzing the results of an SEO experiment. Here are just a few of those options.
- Position in rankings. The first outcome you can look at when you are analyzing the results of an SEO test is the actual search engine rankings for your URLs. For example, you might look at the average position for each URL. While this is often the first outcome that comes to mind, there are some caveats about this outcome that you need to keep in mind. The first is that it is possible for the number of searches that a page is showing up for to dramatically increase or decrease without there being any impact to the average position. That means that there may be important changes that are masked by this metric.
- Number of impressions. It is also possible to look at the number of impressions that a URL gets, but this metric has the opposite problem. It is possible for the number of impressions that a URL is getting to remain constant while the position where that URL shows up in the rankings increases or decreases. The position of the URL in the search rankings will impact the click through rate of the article, but changes in the position will not be reflected in this metric. Therefore, this metric also masks important changes.
- Organic levels. Another possible outcome metric you can look at is the volume of visitors that are clicking into a URL from search engines, or the organic traffic levels that a page gets. This metric accounts for both changes in the position where a URL shows up in the search rankings as well as the number of impressions that a URL gets. If a page shows up higher in the search engine rankings, there will be a higher click thru rate and organic traffic levels will increase. Similarly, if the number of searches where a URL shows up increases, there will be more opportunities for visitors to click on that URL and organic traffic levels will increase. The main consideration to keep in mind when using this metric is that it is possible that a single URL or small subset of URLs with much higher traffic levels than other pages can dominate these metrics.
What methods to use to analyze SEO experiments
How do you analyze data that has been collected in a SEO experiment? Here are some examples of methods that can be used to analyze data from SEO experiments.
- Simple pre-post analysis. The most straightforward way to analyze data from an SEO experiment is to do a simple pre-post analysis. The first step in conducting a pre-post analysis is to calculate the aggregate outcome metric for both the pre-treatment period and the post-treatment period. Do this for both the treatment group and the control group. After that, you can look at the difference between the pre-treatment outcome metric and the post-treatment outcome metric for each test group. The difference between the pre-treatment outcome metric and the post-treatment outcome metric in the treatment group will tell you the difference you would expect to see if your treatment had no effect at all. You can compare that to the difference you see between the pre-treatment value and the post-treatment value in the treatment group to determine whether that difference is larger than the baseline difference from the control group.
- Causal inference. If you want to move beyond a simple pre-post analysis, there are multiple packages that specialize in causal inference methods that can be used to analyze data from SEO tests. One common package that is used to analyze this type of data is the CausalImpact package in R, which uses Bayesian Structural Time Series models.
What to test with SEO experiments
What kinds of changes should you test with SEO experiments? In general you should use SEO experiments to test changes that are intentionally made to impact search engine rankings or large scale changes that you suspect may impact search engine rankings.
Here are some examples of different types of changes you might test using a SEO test.
- Systematic changes to title tags and meta descriptions across many URLs
- Systematic changes to page titles across many URLs
- Systematic changes to headers across many URLs
- Systematic content structure across many URLs
- Systematic changes to internal linking structure
Tips for designing SEO experiments
Here are some final tips we have for designing experiments that test the impact of a change on search engine rankings.
- Isolate changes. Analyzing the results of SEO experiments is nuanced and can be difficult even in the most simple scenario. In order to avoid making the analysis any more difficult, you should isolate the changes that you want to make to your website and test them one at a time. It is okay to run two different hypotheses in two separate tests that are run on entirely different groups of URLs, but you should avoid introducing additional complexity into a single test by trying to test multiple changes at once.
- Have a rollback plan. Sometimes changes that are made in an effort to increase search engine rankings can have sharp negative impacts on rankings. Whenever you test a new change to your website, you should have a rollback plan in mind that will allow you to undo the change swiftly if necessary.
- Avoid low traffic pages. It is generally best to exclude low traffic pages that get no traffic or very minimal amounts of traffic from your SEO tests. If you are using organic traffic levels as the primary outcome for your experiment (which we highly recommend), pages that have very low traffic levels will barely contribute to your outcome metric at all.
- Avoid high traffic pages. Similarly, you should avoid including pages that have extremely high traffic levels from your SEO experiment. If you are using organic traffic levels as the primary outcome for your experiment and you include one page that has tens or hundreds of times more traffic than the others, the traffic from that page will dominate your outcome metric.
- It may take a while for search engine rankings to adjust. The amount of time it takes for a search engine to identify that a change has been made to a website and adjust the search rankings accordingly can vary depending on a variety of factors. You need to plan the timelines for your experiment accordingly and allow for there to be an intermediate period of time between when the treatment is applied and the impact on search rankings is realized.
- SEO test changes that are not intended to have an impact on search engine rankings. Just because a change is not made with the intent of impacting search engine rankings, does not mean that you should not test the impact that change has on your website. It sometimes makes sense to test large scale changes that you are making to your website in an SEO experiment just to ensure that the changes will not have a negative effect on search engine rankings.
