When to use LASSO

Are you wondering when you should use a LASSO model rather than a ridge regression model? Or maybe you are trying to decide whether to use a LASSO model or a standard regression model? Well either way, we’ve got you covered!

In this article we tell you everything you need to know to understand when to use a LASSO model. We start out with a discussion of what types of outcomes LASSO models can handle. After that, we provide more details about the main advantages and disadvantages of LASSO models. Finally, we provide some specific examples of scenarios where you should and should not use LASSO models.

What kind of outcomes can LASSO handle

What kinds of outcome variables can LASSO models handle? One thing to keep in mind when thinking about the kind of outcomes that LASSO models can handle is that the term “LASSO” does not necessarily refer to one unique model. Instead, it refers to a family of models that arises when you introduce a L1 penalty into a standard regression model.

What does this all mean? It means that there are few different types of LASSO models, each of which handle different types of outcome variables.

Can a LASSO model be used with a continuous outcome?

Can a LASSO model be used with a continuous outcome? Yes, there is a LASSO model that can be used with a continuous outcome. In fact, this is probably the most common type of LASSO model out there. This is the model that arises when you take a standard linear regression model and introduce a L1 penalty.

If someone refers to a single type of model using the term “LASSO”, they are almost certainly referring to this model that is derived from a standard linear regression model.

Can a LASSO model be used with a binary outcome?

Can a LASSO model be used with a binary outcome? This is another common question that arises because there are a lot of binary classification problems out there. There is indeed a LASSO model that can be used with a binary outcome.

Just as the standard regression model for binary outcomes is referred to as a “logistic” regression model, the LASSO model for binary outcomes is often referred to as a “logistic LASSO” regression model.

Advantages and disadvantages of LASSO

What are some of the main advantages and disadvantages of a LASSO regression model? Here are some of the main advantages and disadvantages of a LASSO model.

Advantages of LASSO regression

• Automatic features selection. The main advantage of a LASSO regression model is that it has the ability to set the coefficients for features it does not consider interesting to zero. This means that the model does some automatic feature selection to decide which features should and should not be included on its own.
• Reduced overfitting. Another advantage of a LASSO regression is that the L1 penalty that is added to the model helps to prevent the model from overfitting. This makes intuitive sense because when the model sets feature coefficients to zero and effectively removes features from the model, model complexity decreases.

Disadvantages of LASSO regression

• Biased coefficients. One of the main disadvantages of LASSO regression is that the coefficients that are produced by a LASSO model are biased. The L1 penalty that is added to the model artificially shrinks the coefficients closer to zero, or in some cases, all the way to zero. That means that the coefficients from a LASSO model do not represent the true magnitude of the relationship between the features and and the outcome, but rather a shrunken version of that magnitude.
• Difficult to estimate standard errors. Since the coefficient estimates in a LASSO model are biased, it is difficult to estimate accurate standard errors for them. This makes it difficult to do things like run statistical tests on them and build confidence intervals around them.
• Struggle with correlated features. Another downside of LASSO models is that they are very unstable when trained on data with correlated features. What usually happens is that one of the features gets selected somewhat arbitrarily and all of the other features that are highly correlated with that feature get effectively dropped from the model. This may lead someone to erroneously conclude that only the feature that was selected to remain in the model is important, when in reality some of the other features may be just as important or even more important.
• Generally unstable estimates. The estimates produced by LASSO models are known to be relatively unstable, which means that they can change a lot when trained on slightly different datasets. For example, if you bootstrap your data a few times to create a few different sample datasets, you might expect to see that different features get dropped from the model for each dataset. This can happen even if all of the datasets you are training on are very similar.
• Introduction of a hyperparameter. This disadvantage is more minor, but there is a hyperparameter that is introduced in LASSO models to regulate the size of the L1 penalty. That means that you have to go through hyperparameter tuning steps that you would not otherwise have to go through with a standard regression model.
• Other issues associated with standard regression models. LASSO regression models are also plagued by some of the same issues that affect standard regression models. Concerns surrounding interactions, outliers, and stringent model assumptions also apply to this family of models.

When to use LASSO

So when should you use a LASSO regression model? Here are some examples of scenarios when you should use a LASSO regression model.

• Quick and dirty feature selection. LASSO models are usually used to get a quick idea of which features are important for predicting the outcome variable. A LASSO model is trained on all of the features in the data and the features with coefficients that get set to zero are regarded as “not important”. This method works for a quick and dirty analysis, but if you are looking for something more bulletproof, you should keep the instability of LASSO coefficient estimates in mind and consider using other feature selection methods.

When not to use LASSO

When should you not use a LASSO model? Here are some examples of cases where you should avoid using a LASSO model.

• Inference. You should generally avoid using a LASSO model if your primary objective is inference. This is particularly true if you want to be able to determine statistical significance or if you need a precise estimate of the magnitude of the relationship between features and the outcome variable. In these cases, you are better off using a feature selection technique to determine which features are most important and then feeding the selected features into a standard regression model. You can even use a LASSO model for the feature selection step in a pinch.
• Many correlated features. Another situation where you should avoid using LASSO models is if you have many correlated features. If you do find yourself in a situation where you have many correlated features, you should consider using another penalized regression model such as an elastic net model or a ridge regression model instead.

Related articles

• When to use ridge regression
• When to use random forests
• When to use logistic regression
• When to use ordinal logistic regression
• When to use multinomial regression
• When to use support vector machines
• When to use gradient boosted trees
• When to use linear regression
• When to use Bayesian regression
• When to use neural networks
• When to use mixed models

Are you trying to figure out which machine learning model is best for your next data science project? Check out our comprehensive guide on how to choose the right machine learning model.