One of the simplest supervised models are linear models. A linear model expresses the target output value in terms of a sum of weighted input variables. For example, our goal may be to predict the market value of a house, it's expected sales price in the next month, for example. Suppose we're given two input variables. How much tax the property is assessed each year by the local government and the age of the house in years. You can imagine that these two features of the house would each have some information that's helpful in predicting the market price because in most places there's a positive correlation between the tax assessment on a house and its market value. Indeed, the tax assessment is often partly based on market prices from previous years. Maybe a negative correlation between its age in years and the market value, so older houses may need more repair and upgrading, for example. One linear model, which I've made up as an example, could compute the expected market price in US dollars by starting with a constant term here, 212,000, and then adding some number, let's say 109 times the value of tax paid last year, and then subtracting 2,000 times the age of the house in years. For example, this linear model would estimate the market price of a house where the tax assessment was $10,000 and that was 75 years old as about $1.2 million. Now, I just made up this particular linear model myself as an example, but in general, when we talk about training a linear model, we mean estimating values for the parameters of the model or coefficients of the model, as we sometimes call them, which are here, the constant value 212,000 and the weights 109 and 2,000. In such a way that the resulting predictions for the outcome variable Y price for different houses are a good fit to the data from actual past sales. We'll discuss what good fit means shortly. Predicting house price is an example of a regression task using a linear model called not surprisingly, linear regression. While generally in a linear regression model, there may be multiple input variables or features, which will denote, x_0, x_1, etc. Each feature x_i has a corresponding weight w_i. The predicted output, which we denote y hat, is a weighted sum of features plus a constant term b hat. I've put a hat over all the quantities here that are estimated during the aggression training process. The w hat and b hat values, which we call the train parameters or coefficients, are estimated from training data, and y hat is estimated from the linear function of input feature values and the train parameters For example, in the simple housing price example we just saw, w_0 hat was 109, x_0 represented tax paid, w_1 hat was negative 20, x_1 was house age, and b hat was 212,000. We call these w-i values model coefficients or sometimes feature weights, and b hat is called the bias term or the intercept of the model. Here's an example of a linear regression model with just one input variable or feature x_0 on a simple artificial example dataset. The blue cloud of points represents training set of x_0, y pairs. In this case, the formula for predicting the output y hat is just w_0 hat times x_0 plus b hat, which you might recognize as the familiar slope intercept formula for a straight line, where w_0 hat is the slope, and b hat is the y-intercept. The gray and red lines represent different possible linear regression models that could attempt to explain the relationship between x_0 and y. You can see that some lines are a better fit than others. The better fitting models capture the approximately linear relationship where as x_0 increases, y also increases in a linear fashion. The red line seems especially good. Intuitively, there are not as many blue training points that are very far above or very far below the red linear model prediction. Let's take a look at a very simple form of linear regression model that just has one input variable or feature to use for prediction. In this case, we have the vector x is just as a single component. We'll call it x_0. That's the input variable, the input feature. In this case, because there's just one variable, the predicted output is simply the product of the weight w_0 with the input variable x_0 plus a bias term b. X_0 is the value that's provided and comes with the data, and so the parameters we have to estimate are w-0 and b in order to obtain the parameters for this linear regression model. This formula may look familiar. It's the formula for a line in terms of its slope. In this case, slope corresponds to the weight w_0 and b corresponds to the y-intercept, we call the bias term. Here, the job of the model is to take as input, let's pick a point here along the x-axis. W_0 corresponds to the slope of this line, and b corresponds to the y-intercept of the line. By finding these two parameters together to find a straight line in this feature space. Now, the important thing to remember is that there's a training phase and a prediction phase. The training phase using the training data is what we'll use to estimate w_0 and b. One widely used method for estimating w and b for linear regression problem, it's called least-squares linear regression, also known as ordinary least-squares. Least-squares linear regression finds the line through this cloud of points that minimizes what is called the mean squared error of the model. The mean squared error of the model is essentially the sum of the squared differences between the predicted target value and the actual target value for all the points in the training set. This plot illustrates what that means. The blue points represent points in the training set. The red line here represents the least-squares model that was found through this cloud of training points. These black lines show the difference between the y value that was predicted for a training point based on its x position and the actual y value of the training point. For example here, this point, let's say has an x value of negative 1.75. If we plug it into the formula for this linear model, we get a prediction here at this point on the line which is somewhere around, let's say 60. But the actual observed value in the training set for this point was maybe closer to 10. In this case, for this particular point, the squared difference between the predicted target and the actual target would be 60 minus 10 squared. We can do this calculation for every one of the points in the training set. We can compute this squared difference between the y value we observe in the training set for a point and the y value that we'd be predicted by the linear model given that training points x value. Each of these can be computed as this squared difference can be computed. Then if we add all these up, and divide by the number of training points, take the average, that will be the mean squared error of the model. The technique of least-squares is designed to find the slope, the w value, and the b value, the y-intercept that minimize this mean squared error. One thing to note about this linear regression model is that there are no parameters to control the model complexity. No matter what the value of w and b, the result is always going to be a straight line. This is both a strength and a weakness of the model as we'll see later. When you have a moment, compare this simple linear model to the more complex regression model learned with k-nearest neighbors regression on the same dataset. You can see that linear models make a strong prior assumption about the relationship between the input x and output y. Linear models may seem simplistic, but for data with many features, linear models can be very effective and generalize well to new data beyond the training set. Now the question is, how exactly do we estimate the linear models, w and b parameters so the model is a good fit? Well, the w and b parameters are estimated using the training data, and there are lots of different methods for estimating w and b depending on the criteria you'd like to use for the definition of what a good fit to the training data is and how you want to control model complexity. For linear models, model complexity is based on the nature of the weights w on the input features. Simpler linear models have a weight vector w that's closer to zero, i.e., where more features or they're not used at all and have zero weight or have less influence on the outcome with very small weight. Typically, given possible settings for the model parameters, the learning algorithm predicts the target value for each training example, and then computes what is called a loss function for each training example. That's a penalty value for incorrect predictions. Predictions incorrect when they predict the target value is different than the actual target value in the training set. For example, a squared loss function would return the squared difference between the target value and the actual value as the penalty. The learning algorithm then computes or searches for the set of w, b parameters that minimize the total of this loss function over all training points. The most popular way to estimate w and b parameters is using what's called least squares linear regression or ordinary least-squares. Least-squares finds the values of w and b that minimize the total sum of squared differences between the predicted y value and the actual y value in the training set. Or equivalently, it minimizes the mean squared error of the model. Least-squares is based on the squared loss function mentioned before. This is illustrated graphically here where I've zoomed in on the left lower portion of the simple regression data set. The red line represents the least squares solution for w and b through the training data. The vertical lines represent the difference between the actual y value of a training point, x_i [inaudible] y and its predicted y-value given x_i which lies on the red line where x equals x_i. Adding up all the squared values of these differences for all the training points gives the total squared error. This is what the least squares solution minimizes. Here there are no parameters to control model complexity. The linear model always uses all of the input variables and always is represented by a straight line. Another name for this quantity is the residual sum of squares. The actual target value is given in y_i and the predicted y-hat value for the same training example is given by the right side of the formula using the linear model with parameters w and b. Let's look at how to implement this in scikit-learn. Linear regression in scikit-learn is implemented by the linear regression class in the sklearn.linear model module. As we did with other estimators in scikit-learn, like the nearest neighbors classifier and the regression models, we use the train test split function on the original data set and then create an fit the linear regression object using the training data in x_train and the corresponding training data target values in y train. Here, note that we're doing the creation and fitting of the linear regression object in one line by chaining the fit method with the constructor for the new object. The linear regression fit method acts to estimate the feature weights w, which it calls the coefficients of the model. It stores this in the coef_ attribute and the bias term b, which is stored in the intercept_ attribute. Note that if a scikit-learn objects attribute ends with an underscore, this means that these attributes were derived from training data and not say quantities that were set by the user. If we dump the coef_ and intercept_ attributes for this simple example, we see that because there's only one input feature variable, there's only one element in the coef_ list, the value 45.7. The intercept attribute has a value of about 148.4. We can see that indeed these correspond to the red line shown in the plot which has a slope of 45.7 and a y-intercept of about 148.4. Here's the same code in the notebook with additional code to score the quality of the regression model in the same way that we did for k nearest neighbors regression using the R-squared metric. Here's the notebook code we use to plot the least-squares linear solution for this data set. Now that we've seen both K-Nearest Neighbors regression and least squares regression. It's interesting now to compare the least squares linear regression results with the K-nearest neighbor results. Here we can see how these two regression methods represent two complementary types of supervised learning. The K nearest neighbor regressor doesn't make a lot of assumptions about the structure of the data, and it gives potentially accurate but sometimes unstable predictions that are sensitive to small changes in the training data. That has a correspondingly higher training set R-squared score compared to least squares linear regression. K-NN achieves an R-squared score of 0.72 and least-squares achieves an R-squared of 0.679 on the training set. On the other hand, linear models make strong assumptions about the structure of the data. In other words, that the target value can be predicted using a weighted sum of the input variables and linear models gives stable but potentially inaccurate predictions. However, in this case, it turns out that the linear model's strong assumption that there's a linear relationship between the input and output variables happens to be a good fit for this data set. It's better at more accurately predicting the y value for a new x values that weren't seen during training. We can see that the linear model, it gets a slightly better test set score of 0.492 versus 0.471 for K-nearest neighbors. This indicates its ability to better generalize and capture this global linear trend.
