Skip to main content

Section 2.2 Linear Regression - Best Fit Line

In the next few sections, we will presume only one independent variable x and one dependent variable y. Toward that end, consider a collection of data points

\begin{equation*} (x_0,y_0), (x_1,y_1), ... , (x_n,y_n) \end{equation*}

and a general linear function

\begin{equation*} f(x) = mx + b. \end{equation*}

It is possible but generally unlikely that each of the given data points will be interpolated exactly by the linear function. However, you may notice that the data points exhibit a linear tendency or that the underlying physics might suggest a linear model. A "scatter plot" of a example data set is created in teh interactive cell below and the provided data appears to indicate a linear trend. In general, if this is the case then you may find it easier to predict values of y for given values of x using a linear approximation. That is why this method for doing so is also often called a "best-fit line".

But why even bother creating a formula (a line here) to approximate data that does not satisfy that formula? Remember that you would expect collected data to vary slightly as one repeatedly collects that data in the same way that you would expect to make a slightly different score on repeated attempts at exams on the same material. Creating a formula that is close to your data gives a well-defined way to predict a y value for a given x value. This predictive behavior is illustrated in the exercise below.

To determine this best-fit line, you need to determine what is meant by the word "best". For linear regression, to reach this goal consider the total of all vertical deviations between the desired line and the provided data points. Indeed, this vertical error would be of the form

\begin{equation*} e_k = f(x_k) - y_k \end{equation*}

and would be zero if f(x) exactly interpolated at the given data point. Note, some of these errors will be positive and some will be negative. To avoid any possible cancellation of errors, we could consider taking absolute values (which is tough to deal with algebraically) or perhaps squaring the errors. This second option is the standard approach. This approach is similar to the approach taken earlier when developing formulas for the variance.

The best-fit line therefore will be the line \(f(x) = mx+b\) so that the "total squared error" is minimized. This total squared error is given by

\begin{equation*} TSE(m,b) = \sum_{k=1}^n e_k^2 = \sum_{k=1}^n (f(x_k) - y_k)^2 = \sum_{k=1}^n (m x_k + b - y_k)^2. \end{equation*}

For the following interactive cell, consider for the given data points various values for the slope and y-intercept and see if you can make the total squared error as small as possible. In doing so, notice the vertical distances from the line to the given data points generally decreases as this error measure gets smaller.

Experiment in the interative cell above using exactly two data points that have the same x-value. Such as (1,1) and (1,2). Next, add some additional data points in the same general vicinity as your original two points. What is the effect to your best-fit line of adding non-functional points?

So that we don't have to guess the best values for slope and intercept, we can appeal to calculus. Indeed, to minimize this function of the two variables m and b take partial derivatives and set them equal to zero to get the critical values:

\begin{equation*} TSE_m = \sum_{k=1}^n 2(m x_k + b - y_k) \cdot x_k \end{equation*}

and

\begin{equation*} TSE_b = \sum_{k=1}^n 2(m x_k + b - y_k) \cdot 1 . \end{equation*}

Setting each of these equations equal to zero (using calculus!) and solving gives what is known as the "normal equations":

\begin{equation*} m \sum_{k=1}^n x_k^2 + b \sum_{k=1}^n x_k = \sum_{k=1}^n x_k y_k \end{equation*}

and

\begin{equation*} m \sum_{k=1}^n x_k + b \sum_{k=1}^n 1 = \sum_{k=1}^n y_k. \end{equation*}

Notice that these normal equations are a linear system of equations and (among perhaps other reasons) is why this is called linear regression. Solving these for m and b gives the best fit line.

So, let's see how to graph points against the best-fit line using R

Ok. Let's see if you can apply this to get a best fit line.