Why machine learning projects fail

Share this article

Do you want to learn why machine learning projects fail? Well then you are in the right place! In this article, we discuss some of the most common reasons that machine learning projects fail. In particular, we focus on failure modes that are unique to machine learning projects. We do not spend as much time on failure modes that are common across many different industries.

We break this discussion of why machine learning models fail up into three sections. Each of these sections represents a different phase of the project lifecycle where a machine learning project might fail. First, we talk about reasons why machine learning projects fail before a model is created. After that, we talk about reasons why machine learning projects fail after a model is created but before a model is deployed. Finally, we talk about reasons why machine learning projects fail after a model is deployed. For each failure mode we discuss, we also provide tips on how to avoid this failure mode, or at least how to see it coming earlier.

Why machine learning projects fail before a model is created

First, we will talk about why machine learning projects fail before a machine learning model is even created. When we say that these projects fail before a model is created, we do not necessarily mean that they fail before any type of model is ever built. Instead, we mean that these projects fail before any serious model that anyone would consider using to make business decisions is built. It is possible that very simplistic models might have been used for strictly exploratory purposes before this point.

Data is biased or not collected correctly

The first reason that a machine learning project might fail before a model is built is if the data that was intended to be used for the project was not collected correctly. If the flaws that exist in the data collection process are small, there may be techniques that can be used to adjust for biases and render the data usable. If the flaws are large or more widespread, it may be unfeasible to create a machine learning model using that data.

What are some examples of cases where incorrect data collection may lead a machine learning project to fail? One simple example would be if one wanted to build a binary classifier to determine whether an individual belonged to one class or another and data was only collected on individuals that belonged to one of the two classes. In this case, there would be no data on one of the two classes, so there would be no way to determine what individuals that belong to that class do or do not have in common. To some, this situation may seem inconceivable, but these kinds of situations can arise when the data team is not brought into the loop until after all of the data has already been collected.

Data cannot be used for intended purposes

Another reason that a machine learning project might fail before a model is built is because the data that was intended to be used cannot actually be used for the intended purposes. When this situation occurs, it is often because there are laws and regulations that prevent certain types of data from being used for certain purposes. Sometimes the implications of these regulations are not understood until the machine learning project is already underway.

These types of situations where project teams find out that they are not allowed to use the data that they were planning on using happen most often when dealing with sensitive data. This includes personally identifiable information that can be used to identify an individual such as a legal name, address, phone number, or email address. When using this type of information, you should ensure that the data can be used for the intended purpose before investing resources into the project.

Data does not contain any signal

Even if the data has been collected in an appropriate manner and the data is allowed to be used for the intended purposes, there is no guarantee that the data will contain the signal that is required to build a machine learning model. Sometimes, even if you have all of the data that you thought you needed, you might simply find that the data does not contain any signal. This is one factor that makes machine learning projects particularly unpredictable.

Simple exploratory data analysis can be used to help identify situations where there is no signal in the data. For example, imagine you are building a supervised learning model and you do some exploratory analysis to look at bivariate correlations between your features and your outcome variable. Suppose you see that there is absolutely no correlation between the features and the outcome variable. This might indicate that there is no signal in the data, especially if you used nonparametric correlation values that can account for nonlinear relationships between variables. You often need to do some exploratory analysis to determine whether a machine learning project is feasible for this reason.

Why machine learning projects fail after a model is created

What do machine learning projects fail after a machine learning model has already been created? In this section, we will discuss reasons that machine learning projects might fail after a model is created. These are reasons that might crop up after a stable model that the data team has reasonable confidence in is created, but before a model is deployed or the final results are made available.

Unrealistic expectations of predictive performance

One way that machine learning projects can fail after a model has been created is if the business has unrealistic expectations for the predictive performance of the model. There are some situations where business stakeholders want a model to have perfect accuracy or nearly perfect accuracy before they will use it. While in some situations this may truly be required, there are many situations this level of accuracy is not truly required. Even if you have built a successful model, you may be blocked from deploying it if the predictive performance is not considered sufficient.

In general, you should avoid committing to certain levels of predictive performance before a model is created. This is particularly true if there is no strong business logic that can be used to justify the level of predictive performance that is required. If you do have to commit to a certain level of predictive performance, you should make sure to set realistic expectations that are based on concrete evidence (such as the results of an exploratory analysis).

Large amount of computational resources needed

Another reason that machine learning projects can fail after a model has been successfully created is if the model requires a large amount of computational resources. This can occur in cases where a model takes a large amount of resources to train or when a model takes a large amount of resources to make predictions. If the cost of the computational resources required to use the model is greater than the cost that would be saved by using the model, then the machine learning model will likely not be deployed to production.

If you are afraid that you might run into this situation, you should be sure to use simple models that do not require a lot of computational resources whenever possible. You should not spring for more complex models that require a large amount of computational resources until you have validated that simple models are not sufficient for your use case.

Prediction time does not meet latency requirements

Another reason that machine learning projects can fail even after a model has successfully been trained is if the model takes too long to produce predictions and does not meet the latency requirements of the use case. This situation commonly occurs with machine learning models that are going to be incorporated directly into user facing products. If a machine learning model is going to cause a large delay and prevent a user from progressing throughout an experience, that model generally should not be incorporated into the experience.

The best way to avoid these situations is to use simple models whenever possible. Simple models with fewer parameters that need to be calculated and fewer features that need to be accounted for tend to be able to produce predictions faster than complex models with many features and parameters.

Why machine learning projects fail after a model is deployed

Even if a machine learning model is trained and successfully deployed to production, it is still possible for the machine learning project to fail. In this section, we will discuss ways that machine learning projects can fail even after a machine learning model is deployed to production.

Results are not used by stakeholders

The first way that a machine learning project can fail even after a model is deployed to production is if the results of the model are not used by stakeholders. If a model is deployed to production, but the results of the model are not used downstream, the model is not providing any benefit to the company. In this situation, the machine learning project would generally be considered a failure because it demonstrates that the team did not build something that is useful.

The best way to avoid this situation is to ensure that you are building a model that is useful for your stakeholders before you ever build the model. Take the time to understand the requirements for the project and ensure that there is alignment between the data team and the team that will be using the final product.

Model does not affect business outcomes

Even if the results of the model are used by stakeholders, the machine learning project might still be considered a failure if the model does not affect business outcomes. If the results of the model do not unlock new functionalities that impact business metrics, then the model still did not provide any real value to the company.

In order to avoid this situation, data teams should always be mindful of the business metrics they are trying to move when they start machine learning projects. They should prioritize products that are expected to have a direct effect on important business metrics over projects that are not as closely tied to business metrics. Whenever possible data teams should run simulations or smoke tests to estimate the impact that their model might have on business metrics before they start building a machine learning model.

A simple solution would have had the same impact

Even if a model does impact business metrics, a machine learning project can still be considered a failure if a simpler solution that took less time and resources to create could have provided the same impact. If a simple solution could have achieved the same results in less time, then the team wasted time and resources on building a complex solution that does not have any incremental impact.

In order to avoid this situation, data teams should ensure that they use simple baseline models to benchmark the performance of their machine learning models. If they see that a machine learning model cannot outperform a simple solution, they should reconsider whether a machine learning model is appropriate for the project.