In this article I will be demystifying why some of the co-efficients of the predictor variables in Lasso regression will be equal to zero while in Ridge regression none of the co-efficients of the predictor variables will approach  zero(but will not be equal to zero).

If you are new to Ridge and Lasso Linear regression concepts you can check out this article about regression on Analytics Vidhya.

In short Lasso and Ridge regression are used in case there is multicollinearity and to reduce overfitting , so that some of the co-efficients will die down to zero(Lasso) or will approach zero(Ridge).

But after checking out the cost functions for Lasso and Ridge Regression you may get doubt why one cost function reduces the co-efficients to zero and other approaches to zero.

Cost Function for Lasso Regression:-

screen-shot-2016-12-18-at-6-52-48-pm

Let’s say minimising term as A and constraint term as B.

Cost Function for Ridge Regression:-

Screen Shot 2016-12-18 at 6.53.01 pm.png

Let’s say constraint term as C and as said earlier minimising term as A.

So we need to minimise least squares subject to some constraints in Lasso and Ridge Regression. Let us solve this using some graphs.

Let us use two predictor variables, beta1 and beta2 are their co-efficients in regression.

B follows a diamond shaped graph and C follows a circle shaped graph, minimising term A follows a ellipse.

Screen Shot 2016-12-18 at 7.04.39 pm.pngGraph of cost function for Lasso Regression(Figure1)

Screen Shot 2016-12-18 at 7.04.49 pm.png

Graph of cost function for Ridge Regression(Figure2)

As you can see in Lasso the minimising function and constraint can intersect on any one of the axes making their co-efficient values zero while in Ridge the minimising function and constraint cannot intersect on any one of the axes so the co-efficient values will approach zero but will not be equal to zero.

Reference: http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf

 

 

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s