Linear Algebra, Topic: Line of Best Fit

Scientists are often presented with a system that
has no solution and they must find an answer anyway.
That is, they must
find a value that is as close as possible to being an answer.

For instance,
suppose that we have a coin to use in flipping.
This coin has some proportion m of heads
to total flips, determined by how it is
physically constructed, and we want to know if m is near {}1 / 2.
We can get experimental data by flipping it
many times.
This is the result a penny experiment, including some
intermediate numbers.

Because of randomness, we do not find the exact proportion
with this sample — there is no solution to this system.

\begin{matrix}30 m & = & 16 \\ 60 m & = & 34 \\ 90 m & = & 51\end{matrix}

That is, the vector of experimental data is not in the subspace
of solutions.

( \begin{matrix}16 \\ 34 \\ 51\end{matrix} ) ∉ \{ m ( \begin{matrix}30 \\ 60 \\ 90\end{matrix} ) \ | \ m ∈ ℝ \}

However, as described above, we want to find the m that most nearly works.
An orthogonal projection of the data vector into the line subspace
gives our best guess.

\frac{( \begin{matrix}16 \\ 34 \\ 51\end{matrix} ) ⋅ ( \begin{matrix}30 \\ 60 \\ 90\end{matrix} )}{( \begin{matrix}30 \\ 60 \\ 90\end{matrix} ) ⋅ ( \begin{matrix}30 \\ 60 \\ 90\end{matrix} )} ⋅ ( \begin{matrix}30 \\ 60 \\ 90\end{matrix} ) = \frac{7110}{12600} ⋅ ( \begin{matrix}30 \\ 60 \\ 90\end{matrix} )

The estimate (m = 7110 / 12600 ≈ 0.56) is a bit high but not much,
so probably the penny is fair enough.

The line with the slope m ≈ 0.56
is called the line of best fit for this data.

Minimizing the distance
between the given vector and the vector used as the right-hand side
minimizes the total of these vertical lengths,
and consequently
we say that the line has been
obtained through fitting by least-squares

(the vertical scale here has been exaggerated ten times
to make the lengths visible).

We arranged the equation above so that the line
must pass through ( 0 , 0 )
because we take it to be (our best guess at)
the line whose slope is this coin's true proportion
of heads to flips.
We can also handle cases where the line need not
pass through the origin.

For example, the different denominations of U.S. money have different average
times in circulation
(the $2 bill is left off as a special case).
How long should we expect a $25 bill to last?

The plot (see below) looks roughly linear.
It isn't a perfect line, i.e.,
the linear system with equations b + 1 m = 1.5, ..., b + 100 m = 20 has no
solution, but we can again use orthogonal projection
to find a best approximation.
Consider the matrix of coefficients of that linear system and also its
vector of constants, the experimentally-determined values.

A = ( \begin{matrix}1 & 1 \\ 1 & 5 \\ 1 & 10 \\ 1 & 20 \\ 1 & 50 \\ 1 & 100\end{matrix} ) \ \overset{→}{v} = ( \begin{matrix}1.5 \\ 2 \\ 3 \\ 5 \\ 9 \\ 20\end{matrix} )

The ending result in the subsection on Projection into a Subspace says
that coefficients b and m so that the linear combination
of the columns of A is as close as possible to the vector \overset{→}{v}
are the entries of ( {A}^{t r a n s} A {)}^{− 1} {A}^{t r a n s} ⋅ \overset{→}{v}.
Some calculation
gives an intercept of b = 1.05 and a slope
of m = 0.18.

Plugging x = 25 into the equation of the line shows that such a
bill should last between five and six years.

We close by considering
the times for the men's mile race (Oakley & Baker 1977).
These are the world records that were in force on January first
of the given years.
We want to project when a 3:40 mile
will be run.

We can see below that the data is surprisingly linear.
With this input

A = ( \begin{matrix}1 & 1860 \\ 1 & 1870 \\ ⋮ & ⋮ \\ 1 & 1990 \\ 1 & 2000\end{matrix} ) \ \overset{→}{v} = ( \begin{matrix}280.0 \\ 268.8 \\ ⋮ \\ 226.3 \\ 223.1\end{matrix} )

the Python program at this Topic's end gives

\text{slope} = − 0.35

and \text{intercept} = 925.53
(rounded to two places; the original data
is good to only about a quarter of a second
since much of it was hand-timed).

When will a 220 second mile be run?
Solving the equation of the line of best fit
gives an estimate of the year 2008.

This example is amusing, but serves as a caution — obviously the
linearity of the data will break down someday (as indeed it does
prior to 1860).

Exercises

The calculations here are best done on a computer. In addition, some of the problems require more data, available in your library, on the net, in the answers to the exercises, or in the section following the exercises.

Problem 1

Use least-squares to judge if the coin in this experiment is fair.

Problem 2

For the men's mile record, rather than give each of the many
records and its exact date, we've "smoothed" the data somewhat by
taking a periodic sample.
Do the longer calculation and compare the conclusions.

Problem 3

Find the line of best fit for the men's 1500 meter run.
How does the slope compare with that for the men's mile?
(The distances are close; a mile is about 1609 meters.)

Problem 4
Find the line of best fit for the records for women's mile.

Problem 5

Do the lines of best fit for the men's and women's miles cross?

Problem 6

When the space shuttle Challenger exploded in 1986, one of the
criticisms made of NASA's decision to launch was in the way the
analysis of number of O-ring failures versus temperature was made
(of course, O-ring failure caused the explosion).
Four O-ring failures will cause the rocket to explode.
NASA had data from 24 previous flights.

The temperature that day was forecast to be {}3 {1}^{∘} \text{F}.

• NASA based the decision to launch partially on a chart showing
  only the flights that had at least one O-ring failure.
  Find the line that best fits these seven flights.
  On the basis of this data,
  predict the number of O-ring failures when the temperature is 31, and
  when the number of failures will exceed four.
• Find the line that best fits all 24 flights.

On the basis of this extra data,
predict the number of O-ring failures when the temperature is 31, and
when the number of failures will exceed four.

Which do you think is the more accurate method of predicting?
(An excellent discussion appears in (Dalal, Folkes & Hoadley 1989).)

Problem 7

This table lists the average distance from the sun to
each of the first seven planets, using earth's average as a unit.

• Plot the number of the planet
  (Mercury is 1, etc.) versus the distance.
  Note that it does not look like a line, and so finding the
  line of best fit is not fruitful.
• It does, however look like an exponential curve.

Therefore, plot the number of the planet versus the
logarithm of the distance.
Does this look like a line?

• The asteroid belt between Mars and
  Jupiter is thought
  to be what is left of a planet that broke apart.
  Renumber so that Jupiter is 6, Saturn is 7, and Uranus is
  8, and plot against the log again.
  Does this look better?
• Use least squares on that data to predict the

location of Neptune.

• Repeat to predict where Pluto is.
• Is the formula accurate for Neptune and Pluto?

This method was used to help discover Neptune (although the second item is
misleading about the history; actually, the discovery of Neptune
in position 9
prompted people to look for the "missing planet" in position 5).
See (Gardner 1970)

Problem 8

William Bennett has proposed an Index of Leading Cultural Indicators
for the US (Bennett 1993).
Among the statistics cited are the average daily hours spent watching TV,
and the average combined SAT scores.

Suppose that a cause and effect relationship is proposed between
the time spent watching TV and the decline in SAT scores (in this article,
Mr. Bennett does not argue that there is a direct connection).

• Find the line of best fit relating the independent variable
  of average daily TV hours to the dependent variable of SAT scores.
• Find the most recent estimate of the average daily TV hours

(Bennett's cites Neilsen Media Research as the source of these
estimates).
Estimate the associated SAT score.
How close is your estimate to the actual average?
(Warning: a change has been made recently in the SAT,
so you should investigate whether some adjustment needs to be made to
the reported average to make a valid comparison.)

Solutions

Computer Code

#!/usr/bin/python

data file format: each line is two numbers, x and y

n = 0
sum_x = 0
sum_y = 0
sum_x_squared = 0
sum_xy = 0

fn = raw_input("Name of the data file? ")
datafile = open(fn,"r")
while 1:
ln = datafile.readline()
if ln:
data = ln.split()
x = float(data[0])
y = float(data[1])
n += 1
sum_x += x
sum_y += y
sum_x_squared += x*x
sum_xy += x*y
else:
break
datafile.close()

slope = (nsum_xy - sum_xsum_y) / (n*sum_x_squared - sum_x**2)
intercept = (sum_y - slope*sum_x)/n
print "line of best fit: slope= %f intercept= %f" % (slope, intercept)

Additional Data

Data on the progression of the world's records (taken from the Runner's World web site) is below.

References

• Bennett, William (March 15, 1993), "Quantifying America's Decline", Wall Street Journal{{citation}}: CS1 maint: date and year (link)
• Dalal, Siddhartha; Folkes, Edward; Hoadley, Bruce (Fall 1989), "Lessons Learned from Challenger: A Statistical Perspective", Stats: the Magazine for Students of Statistics, pp. 14–18{{citation}}: CS1 maint: date and year (link)
• Gardner, Martin (April 1970), "Mathematical Games, Some mathematical curiosities embedded in the solar system", Scientific American, pp. 108–112{{citation}}: CS1 maint: date and year (link)
• Oakley, Cletus; Baker, Justine (April 1977), "Least Squares and the 3:40 Mile", Mathematics Teacher{{citation}}: CS1 maint: date and year (link)

Where this page came from

This page was imported from Wikibooks. From “Linear Algebra” on Wikibooks, by its contributors, under CC BY-SA 4.0. Changed here: set as a page; navigation and edit links left out; each image under its own licence, credited in its caption.

Nobody has written it yet — it is the source material at a new address, which is why search engines are asked to skip it and why no one earns from it. It is up for grabs: take it on, and it is yours to rewrite and to earn from.