Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems

Abstract

This report describes a test of the large language model GPT-4 with the
the Wolfram Alpha and the Code Interpreter plug-ins on 105
original problems in science and math, at the high school and college
levels, carried out in June-August 2023.
Our tests suggest that the plug-ins significantly enhance GPT’s
ability to solve these
problems. Having said that, there are still often ”interface”
failures; that is, GPT often has trouble formulating problems in a way
that
elicits useful answers from the plug-ins. Fixing these interface
failures seems like a central challenge in making GPT a reliable tool
for college-level calculation problems.

This report describes a test of the large language model (LLM)
GPT-4 with the plug-ins
Wolfram Alpha (henceforth ‘‘GPT4+WA’’) and Code Interpreter (henceforth
‘‘GPT4+CI’’) on 105 expert-crafted,
original problems in science and math, at the high school and college
levels, carried out in June-August 2023.
Table 1 shows
examples of the problems used: three where both AI systems succeeded and
three where both failed. The complete collection of problems are in appendices
A-C; the complete outcomes of the two AIs are in
tables 3-7.

Successes

Both GPT4+WA and GPT4+CI got the correct answer on these.

Problem, “Arbitrary Numerical” test set: A point p is chosen at random within the 100-dimensional box
{\bf B}=[0,100]^{100} following a uniform distribution.
What is the probability
that the Euclidean distance from p to the boundary of \bf B
is less than 1?

Answer: {}1-0.98^{100}\approx 1-1/e^{2}=0.8647

Problem, “Calculation-free” test set: Let C be the center of the earth.
Can there be three earth satellites X, Y, and Z such that C, X, Y, and Z
are always coplanar?
Answer: Yes.

Problem, “Motivated Numerical” test set:
What is the probability that a randomly-chosen 100 \cdot 100 matrix,
over the finite field F_{2}, is invertible?
Please include all calculations.

Answer

\prod_{i=1}^{100}\frac{2^{j}-1}{2^{j}}\approx 0.289

Failures

Problem, “Arbitrary Numerical” test set:
Viewed from Vega, what is the angle between Sirius and the Sun?

Correct Answer: 0.0972 radians = {}5.6^{\circ}. GPT4+WA:
.005060 degrees.
GPT4+CI: “The angle between Sirius and the Sun as viewed
from Vega depends on the time of year, as well as the specific time of day.
This is because the Earth and Vega both orbit the Sun, but
at different rates, and because the Earth also rotates on its axis. …”

Problem, “Calculation-free” test set:
Joe says that he lives 10 miles from Lake Michigan, that
Beth lives
10 miles from Lake Michigan,
and that he and Beth live 100 miles apart. Is it possible that
Joe is telling the truth? Answer ‘Yes’ or ‘No’.”
Correct Answer: Yes. GPT4+WA and GPT4+CI both answer: No.

Problem, “Motivated Numerical” test set:
What is the Shannon entropy of a positive integer n that’s chosen
with probability Pr[n] = {}6/(\pi^{2}\cdot n^{2})?
Please include all calculations.
Answer: 2.362. GPT4+WA: did not return an answer.
GPT4+CI: 0.9265.

Table 1: Examples of successes and failures on test set problems

1 Summary of Conclusions

Our test sets were too small and too haphazard to support statistically valid
conclusions, but they were suggestive of a number of conclusions. We
summarize these here, and discuss them at greater length
in section 7.

Over the kinds of problems tested, GPT-4 with either plug-in is significantly
stronger than GPT-4 by itself, or, almost certainly, than any AI that existed
a year ago. However it is still far from reliable; it often outputs a wrong
answer or fails to output any answer.

In terms of overall score, we would judge that these systems performs on the
level of a middling undergraduate student. However, their capacities and
weaknesses do not align with a human student; the systems solve some
problems that even capable students would
find challenging, whereas they fail on some problems that even middling
high school students would find easy.

There is considerable room for improvement with the interfaces between GPT-4
and the plug-ins, particularly with Wolfram Alpha. GPT-4 often struggles to
formulate a problem in a way that Wolfram Alpha can accept or that produces
useful output. GPT-4 also sometimes creates quite useless calls to
the plug-ins.

GPT-4 fails to take full advantage of the capacities of the plug-ins,
particularly Wolfram Alpha.
The LLM often carries out calculations, retrieves formulas, and does symbolic
manipulations that the specialized plug-ins could carry out more reliably.
This sometimes leads to errors that should have been avoidable.

Overall, the systems are strongest on problems that can be solved by invoking
a single formula. They are often weak on problems that humans would tend to
solve using spatial visualization. They are often weak on problems that involve
combining several calculations of different kinds. The systems often have
trouble dealing with numbers that are very large or very small.

GPT-4 has some ability to detect when the answer returned by a plug-in is
nonsensical or physically meaningless, but not reliably, and it has little
ability to diagnose the reason for the mistake or to recover from it.

2 The test sets: Overview

We created three test sets of original problems of somewhat different flavors
which we here call the ‘‘Arbitrary Numerical’’ test set, the ‘‘Calculation-Free’’
test set, and the ‘‘Motivated Numerical’’ test set. The problems in the first
two were written by Davis, the third was written by Aaronson; they
reflect the interests and tastes of their authors.
The complete collection
of problems with answers are in appendices A, B, and C below.

2.1 The “Arbitrary Numerical” test set: Overview

The first test set, which for convenience we will call the “Arbitrary
Numerical”
test set, contains 32 problems, whose answers are either numerical or,
in four cases, two- or three-dimensional vectors. Other than numerical
constants, which are easily available in standard references such as Wikipedia,
and are known by both GPT-4 and Wolfram Alpha, the science involved in
this dataset is quite elementary. There are two questions that require using
the relativistic formula for energy and two that involve Planck’s law relating
the wavelength and energy of a photon; otherwise, the science required is
all at the level of high school or first-semester college physics and
chemistry. The mathematics demanded is somewhat more demanding, drawing
on such subjects as three-dimensional geometry, spherical geometry, linear
algebra, probability theory, integral calculus, and, in one problem, asymptotic
analysis; but only very elementary aspects of these fields.
The problems are “arbitrary” in the sense that each problem involves a
fairly random collection of specifics and that the individual problems are
not of any particular inherent interest.
Some of the problems would be considered somewhat complex and, frankly,
rather tedious for a human student, but none involve any very deep reasoning.

2.2 The “Calculation-Free” test set: Overview

The second test set, called
the “Calculation-Free” test set, contains 53 questions with
discrete answers: 36 problems with two possible answers, 10 multiple choice
problems, and 7 sorting problems. Since a reasonable score can be gotten
by guessing, we included multiple instances of problems of the same general
type. In general a human test-taker who has a general knowledge of the
subject matter and who either knows or can look up the geographical and
historical information involved can solve these without using a calculator
or setting pencil to paper.

The first category, “Eclipses” consists of eight
multiple-choice questions about what an astronomer on the
moon would observe during an eclipse. For example:

An astronaut is standing in the Sea of Tranquility during what on earth is called a total lunar
eclipse. They are looking in the direction of the earth. What they see is:
A. The surface of the moon, illuminated by earth light.
B. The night side of the earth, occluding the sun.
C. The surface of the moon, illuminated only by starlight.
D. The surface of the moon, illuminated by the sun.
E. The sun.
F. The day side of the earth, with a small circular shadow moving quickly over it.
G. The night side of the earth. The sun is somewhere else entirely.
H. A starry sky. Neither the sun, the earth, or the surface of the moon is in the field of view.

(The answer is B.)
Anyone with an elementary-school understanding of the relation of the earth,
moon, and sun during eclipses should be able to answer these without
difficulty.

The second category, “Distance combinations”
consists of seven questions along the following lines,
“Joe says that he lives 10 miles from Lake Michigan, that Beth lives
10 miles from Lake Michigan,
and that he and Beth live 100 miles apart. Is it possible that
Joe is telling the truth? Answer ‘Yes’ or ‘No’.”

Anyone who knows, or can look up on a map, the geography of the
Atlantic Ocean, Lake Michigan, Lake Huron, and Walden Pond (the four bodies of
water mentioned) can easily solve these.

The third category, “Distance between points in rivers”, contains five
questions
such as the following: “Is there a point x in the Danube River and a point y in the Rhine River that are exactly 429 miles apart? Answer ‘Yes’ or ‘No’.”

Anyone who is familiar with the geography of the Nile, Danube, and Rhine
rivers, or can look them up on a map, can answer these easily.

The fourth category, “Clockwise or counter-clockwise” presents a triple
of Western Hemisphere cities, such as “Caracas, Venezuela;
Amarillo, Texas; Quebec, Quebec,”
and asks if the sequence is in clockwise or counter-clockwise order.
Anyone who can visualize the geography involved or can look up a map can
answer these easily. There are twelve of these questions.

The fifth category, “Sorting” contains seven problem that ask
the users to sort specified objects or events
(between 8 and 11) in increasing order by mass, temporal characteristics,
or distance from a reference point. For example,

Sort the events below by starting date:
A. The lifetime of Marie Antoinette.
B. The Precambrian period.
C. The first performance of Beethoven’s seventh symphony.
D. Lincoln speaking the Gettysburg address.
E. The Hundred Years’ War.
F. The reign of Queen Victoria.
G. The existence of the species of passenger pigeons (ending with the death of “Martha”).
H. The battle of Gettysburg.
I. The existence of legal slavery in what is now the United States.
J. The reign of Marie Antoinette.
K. The lifetime of Joan of Arc.

Due to a misdesign, three of specific comparisons in the seven problems turned
out to be tricky,
but the rest could be easily judged by someone who knows
or can look up the relevant history and geography.

The sixth category “Satellites”
consists of fourteen problems about the feasibility of
Earth satellite orbits with various characteristics. Twelve of these are binary
choice; two are multiple-choice. For example, “Is it possible to have
a satellite such that the northernmost earth point underneath the
satellite has latitude 40 degrees N and the southernmost has latitude
30 degrees S? Assume that the satellite is moving in a closed orbit
around the Earth and that the only influence on the satellite’s motion
is the Earth’s gravity. Assume that the Earth is a perfect sphere.
Ignore the revolution of the Earth around the sun, but do not ignore
the rotation of the Earth around its axis.”

These require an understanding of the basic properties of satellite orbits and
some ability to reason about three-dimensional geometry. For example, answering
the question above requires understanding that the plane of a satellite’s orbit
always includes the center of the earth, and that that implies that the
the southernmost point will have equal latitudes with opposite signs. However,
no calculations are ever involved.

2.3 The “Motivated Numerical” test set: Overview

The third test set, called the “Motivated Numerical” test set, consists
of 20 problems with numerical answers. These were designed to be more
natural and of greater inherent interest than the “Arbitrary Numerical”
problems; one can imagine someone raising one of them in casual geeky
conversation as an interesting factoid, which is rarely the case for the
arbitrary problems. They draw on a wide range of areas in math and physics,
often somewhat more advanced than the arbitrary problems.

3 History of the AI systems and the testing project

AI technology currently advances (and occasionally retreats)
so rapidly that it has become necessary to
report precise dates for experiments, and that even quite short and small
projects can be upended in midstream by new external developments.

On March 14, 2023, OpenAI (2023a, 2023b) released the LLM GPT-4.
On March 23, OpenAI (2023c)
that GPT-4 would support plug-ins. The same day Steven Wolfram (2023)
announced the availability of a plug-in that would allow GPT-4 to
interface with the ‘‘answer engine’’ Wolfram
Alpha. The announcement gives
considerable detail and many examples. Plug-ins are currently
considered a beta-level release; they are
available to paying subscribers to ChatGPT-Plus.

It seemed to us likely that the combination of GPT-4 with Wolfram Alpha
might be significantly more powerful for science and math problems posed
in English than earlier systems; it certainly seemed worth testing.
We developed three test sets of original problems, described in
section 2 below, and included in full in
appendices A, B, and C.
We began testing GPT4+WA on June 28. By July 9 we had nearly completed the
testing and analysis of the first two test sets.

On July 9 we learned that, three days earlier, on July 6, OpenAI had
announced
the beta-level release of
a plug-in “Code Interpreter” which enables GPT-4 to write and execute
code in Python, with access to Python’s extensive mathematical libraries.
It was clear that our
tests should also be run on the GPT4+CI;
we carried out those tests of GPT+CI
on the first two data sets between July 9 and
July 12. The third data set was tested between July 27 and August 2
(table 2).
It is these tests that are described in this report.

Table 2: Dates of experiments

4 The Design and Testing Process

The three sets of questions, and their answers, were all prepared in advance
of any testing; the authors had no experience using GPT-4 with the plug-ins.
Some edits for disambiguation were found to be
necessary in the
course of testing, and consequently some problems were run more than once,
with different wordings. For example: In problem 3 of the Arbitrary Numerical
test set, we initially omitted the unit length of the tetrahedron.
Problem 19 of the Arbitrary Numerical test set was originally worded “…vertex B is
in the x-y plane”; we changed that to “in the x-z plane, with positive
x-coordinate” since otherwise the problem is ambiguous. In the “Satellite”
problems of the “Calculation Free”
test set, we added a specification of the idealizations
to be applied. However, all of these were driven by considerations of
the problems themselves, not in reaction to the results from our tests; and
no contentive changes were made.

One problem was deleted, after running it through the AI systems, because
we discovered that the correct answer is an open problem in coding
theory.

To avoid cross-question contamination, we started a new Chat session for
each of the problems. The only exception was the “clockwise” problems,
where that seemed burdensome and unnecessarily;
rather, for the first question we wrote,

If you have a map that shows Chicago, New York City, and Atlanta, and
you draw a circle through the three of them, then the sequence
< Chicago; New York; Atlanta > is in clockwise order. The sequence <
New York; Chicago; Atlanta> on the other hand, is in counterclockwise order.

Is the following in counterclockwise or counterclockwise order: Caracas, Venezuela; Amarillo, Texas; Quebec, Quebec.

and then for the subsequent questions we continued the session and asked,

How about New Orleans, Louisiana; Springfield, Illinois; Jacksonville, Florida.

and so on.

Obviously, this collection of problems is quite unsystematic and arbitrary,
as well as small, and it does not support any broad conclusions about the
capabilities, absolute or relative, of these two systems. Even if one
system had achieved a perfect success on these test sets, and the other
had failed abjectly, that would hardly justify concluding that the first
is better in general than the second, let alone that the first “understands”
the subject matter and the second does not. However, the
results are suggestive of some interesting strengths and weaknesses.

We did not experiment with few-shot prompts or “chain of thought” prompts.
(In most cases, the two systems output a chain of thought in any case.)
In many of our tests of the Arbitrary Numerical test set we ended our prompt.
“Please include
all calculations in LaTeX, suitable for copying and pasting;” the system
sometimes complied with that format request, but not at all reliably.
In all of our tests of the “Motivated Numerical” test set, we ended our
prompt with “Please include all calculations”.

5 Results

Tables 3-7
give a tabular summary of the results of our
tests.

We use the error marking “Wrong analysis” in cases where, if a human being
had produced the output, a grader would be inclined to say that they had
understood the problem conceptually but had erred in setting up the mathematical
analysis. We use the error marking “Misunderstood problem” in cases where a
grader would say that a human really did not understand, either what was the
specification or what was being asked. Obviously, the borderline between these
is indeterminate; also obviously, applying them to AIs is reckless
anthropomorphism, meant only to be suggestive.

In the categorization of the science and math required, we focus on the most
important component of the analysis.

Table 3: Results for scientific problems: 1-16

In total, GPT4+WA got a score of 8.25 out of 32, giving partial credit on
three problems; GPT4+CI got a score of 10 out of 32.

Table 4: Results for scientific problems: 17-32

Table 5: Results for calculation-free problems 1-32

Sorting problems are scored in terms of the fraction of correct pairwise
comparisons. In this category, GPT4+WA always called WA to find the quantities,
with disastrous consequences in questions 35 and 37. Our evaluation of the
“explanation” here reflects the accuracy of the AI’s estimation of the
quantities involved, rather than the success in sorting those. (The sorting
was always carried out by GPT rather than the plug-in.)

GPT4+WA: 30.7/53. GPT4+CI: 34.2/53. Guessing randomly: 22.6/53

Table 6: Results for calculation-free problems 35-53

‘

Table 7: Results from the “Motivated Numerical” test set. GPT4+WA: 14.3/20.
GPT4+CI: 13.8/20

6 Related Work

6.1 SciBench

A similar study has been published very recently by Wang et al. (July 2023).
They created a new benchmark collection “SciBench” consisting of
695 problems collected from ten existing college-level textbooks
in physics,
chemistry, and math, 112 of which had detailed step-by-step
solutions and 104 problems from midterms and exams from courses in data mining,
machine learning, and differential equations. All of the textbook problems,
and most though not all of the exam problems, have numeric answers.

The SciBench benchmark was used to test GPT-3.5 and GPT-4 in seven
different prompting protocols: Zero-shot with (1) and without (2) the system
prompt; (3) few shot; (4) chain of thought; (5) few-shot with chain of
thought;
(6) generating executable Python code using few-shot prompting
(7) generating executable Wolfram Alpha code using few-shot prompting.
The strengths and weakness thus revealed were analyzed by a combination
of human annotators and automated analysis using GPT-3.5.

Obviously, this was a much larger undertaking than our experiment; their test
set is eight times larger than ours, and they ran their experiment on fourteen
differenti models, while we ran ours on two.
Obviously,
there is some overlap but there are also significant differences in the
kinds of
problems in our data sets vs. theirs. In particular, SciBench seems to contains
few problems similar to those in our “Calculation-Free” collection and
has a smaller proportion of geometry problems and astronomy problems.
One difference that should
be particularly noted is that, though, like us, Wang et al. tested combinations
of GPT with Python and with Wolfram Alpha, they used their own system of
prompts rather than the plug-ins that have been created by OpenAI and
Wolfram Alpha. That makes a substantial difference. For instance, Wang
et al. report that “[Prompts] that utilize Wolfram diminish performance,”
whereas we found that using the Wolfram Alpha plug-in certainly improved
performance over zero-shot GPT-4.

6.2 Other work

Many benchmark collections of mathematical word problems have been
assembled The mathematical difficulty ranges from
elementary school to International Mathematical Olympiad and introductory
college math courses. The language use ranges from non-linguistic to
moderately complex. Some require no world knowledge; others requires some
combination of commonsense, common, expert, and encyclopedic knowledge.
Question formats include open form question answering,
fill-in-the-blank, multiple choice, and proof construction. Some notable
recent examples: SVAMP (Simple Variations on Arithmetic
Math Problems) (Patel, Bhattamishra, and Gohal, 2021)
is a collection of word problems involving
only one or two arithmetic operations and integers between 1 and 999.
Līla (Mishra et al. 2022) is a wide-ranging collection, assembled by
combining twenty-three earlier problem collections; GSM8K (Cobbe et al.
2021) is a collection of 8500 grade-school math word problems. The
MATH dataset (Hendryks et al. 2021) consists of 12,500 problems
taken from high school competitions. The GHOSTS dataset (Frieder et al., 2023)
has 703 problems taken
from advanced undergraduate and graduate math courses.

Likewise, quite a few science benchmarks have been assembled, covering
a range of forms, fields, and levels. ScienceQA (Lu, 2022), is a collection
of 21,000 multimodal science questions, annotated with explanations.
The Textbook Question Answering (TQA) dataset includes 1100 lessons and 26,000
multi-modal questions taken from middle-school science curricula. The ARC
dataset (Clark et al. 2018) is a collection of 7800 multiple-choice
science questions at a grade school. The BIG-bench collection (Srivastava
et al. 2022) contains 162 benchmarks that are tagged as dealing with
“math, logic, and code” and 20 that are tagged as dealing with “scientific
and technical understanding.”

7 Discussion

Our test sets were too small and too haphazard in construction
to support statistically valid
conclusions, but they were suggestive of a number of conclusions.

In our discussion here, we will refer to the problems in the “Arbitrary”
test set as A.1-A.32; those in the “Calculation-Free” test set as B.1-B-53;
and those in the “Motivated” test set as C.1-C.20.

We feel confident that, over the kinds of problems tested,
GPT-4 with either plug-in is significantly
stronger than GPT-4 by itself, or, almost certainly, than any AI that existed
a year ago. (We do not have experimental evidence for the claim; we carried
out few tests on default GPT-4 and none at all on other systems.)
It is much stronger than Wolfram Alpha, whose natural language interface
is very brittle. These systems may well be the strongest AI systems for
these problem currently in existence.
However they are still far from reliable; they often output a wrong
answer or fail to output any answer.

The absolute numerical scores --- ‘‘GPT-4+WA got 8.25/32 on the Arbitrary
Numerical test set’’ and so on --- are meaningless in themselves, since
the construction and choice of the problems was arbitrary and non-replicable.
Broadly speaking, though, we would guess that the overall performance of
these systems on these test sets was at the level of a middling
undergraduate student provided with the physical constants involved.
We would expect a capable math or physics major in their sophomore year
to get a good score, if they were provided with the physical
constants involved and could use a calculator.
However, the capacities and
weaknesses of these AI systems
do not align with a human student; the systems solve some
problems that even strong students would
find challenging, such as the linear algebra problem C.14 , whereas they
fail on some problems that even middling
high school students should find easy, such as the eclipse problems B.1-B.8.

It would seem, though our tests are too small to establish conclusively,
that the two plug-ins are incomparable in power; each can solve problems
that the other cannot. Neither seems decisively superior overall to the
other.

There is considerable room for improvement with the interfaces between GPT-4
and the plug-ins, particularly with Wolfram Alpha. GPT-4 often struggles to
formulate a problem in a way that Wolfram Alpha can accept or that produces
useful output. For instance, in problem B.35, GPT-4 called Wolfram Alpha 40
times unsuccessfully trying to get the date of the Hundred Years’ War and
then gave up (GPT-4 without plug-ins knows that date in any case.) In problem
C.14, a probabilistic calculation, the answer returned by Wolfram Alpha was
in the form of the exact rational number with a numerator of 1521 digits and a
denominator of 1522 digits. GPT-4 was unable to estimate its value, and it
did not occur to GPT-4 to rephrase the request to Wolfram Alpha to return
the value in an intelligible form. In problem A.14, which asked for an answer
in astronomical units, Wolfram Alpha returned the correct answer expressed
in meters, but then the system was unable to get that converted to astronomical
unites, despite making eight calls to Wolfram Alpha.

GPT-4 sometimes created useless calls to
the plug-ins, which asked for no useful information. For example, in problem
C.11, GPT carried out the following exchange with WA:

Request to Wolfram
{
"input": "F = m*a; F"
}
Response from Wolfram
"a m"

In problem C.15, GPT-4 issued the following call to CI:

pythonCopy code

Total weight of gold and uranium mined (in kg)

gold_weight_kg = 190_000_000 # kg
uranium_weight_kg = 2_500_000_000 # kg

gold_weight_kg, uranium_weight_kg

That is, the plug-ins simply being asked to echo formulas or
values that GPT-4 itself provided.

GPT-4 fails to take full advantage of the capacities of the plug-ins,
The LLM often carries out calculations, retrieves formulas, and does symbolic
manipulations that the specialized plug-ins could carry out more reliably.
This sometimes leads to errors that should have been avoidable.
For instance in problem C.9, GPT4 made a mistake in a symbolic
manipulation of a formula; either WA or sympy can carry out such
operations flawlessly. In problem A.20, which involved computing compound
interest over a number of years, the “natural” solution would be to write
a small for loop to carry out the calculation, and indeed GPT4+WA did
exactly that; however, GPT4+CI attempted to carry out the loop purely in
GPT, without calling CI, which led to an erroneous answer. In the sorting
problems B.33-B.39, GPT4 never called either plug-in to carry out the sorting;
it did the sorting itself, and not always correctly.

Overall, the systems are strongest on problems that can be solved by invoking
a single formula. They are often weak on problems that humans would tend to
solve using spatial visualization, such as the eclipse problems B.1-B.8.
They are often weak on problems that involve
combining several calculations of different kinds, such as the pendulum
problem A.12 or the chemical reaction problem A.24. The systems sometimes have
trouble dealing with numbers that are very large or very small. The
systems seem to have particular difficulty with the theory of special
relativity, when problems go beyond simply plugging in one of the standard
formulas (problems A.3 and C.11).

GPT-4 has some ability to detect when the answer returned by a plug-in is
nonsensical or physically meaningless, but not reliably.
Its performance would be improved were it better able to diagnose
the reasons for the mistakes and to recover from them.

Over the course of these problems, GPT-4 does a fair amount of arithmetic
calculations over floating point numbers without calling either plug-in.
Occasionally it makes a mistake – for instance, in problem B.20 it decided
that $1449.46 is larger than $ 1823.26 —
but much more frequently, it gets the
correct answer. This is quite remarkable in itself.

Both of these systems tend to use “brute force” methods which can involve
unnecessary computation. In particular, they almost always convert quantities
to standard units even when that conversion is unnecessary. For instance,
in problem C.11, both systems used the value for the speed of light, even
though the problem can be solved taking c=1. In problem A.25, dealing with
radioactive decay, GPT4+WA brought in Avogadro’s number, even though
the problem can be solved entirely at the level of masses and moles. On the
other hand, in problem A.5, it found a solution path more elegant than Davis’
and in problem A.25, its solution correctly incorporated a significant
consideration that Davis had overlooked.

It seems likely that GPT4+CI and GPT4+WA are most useful
when not relied on as
“oracles,” but in “interactive mode,” with a human using them to speed
up problem-solving but also making multiple queries, checking that the
outputs look reasonable, and so on. We examined this option only
in two cases, but it deserves further study.

7.1 Methodological observations

We conclude with a couple of methodological observations. First: The methodology
we have used in these tests is, currently, a somewhat unusual one for AI
systems: We have personally constructed a small, idiosyncratic, test set,
we manually examined
the entire output on that set, we carried out a quite painstaking analysis,
and we have published the entire test set with hand-written explanations
of the answers and the entire outputs of both systems, annotated
and lightly edited.
This approach does not yield conclusions that to any degree support statistical
analysis or statistical measures of confidence, and it would expensive to
scale it to a test set large enough to support that analysis.
However, it does enable
the detailed analysis of the strengths and weaknesses of these systems
that we have given in section 7 above.

Second: Our analysis relies heavily on the explanations generated by the
AI systems in their output. In general, that kind of reliance can be
risky in analyzing the behavior of LLMs. In general an LLM generates
an answer to a question by finding the most likely sequence of tokens to
fit the “answer” slot; it generates the explanation of the answer by
finding the most likely sequence of tokens to fit the “explanation” slot;
the explanation may have nothing to do with the way in which it actually
generated
the answer. However, for many of the problems in our test set, that
caution is mitigated. One can be reasonably confident that the explanation
is, in fact, related to the answer for two reasons:

• 1.

In problems where GPT-4 calls the plug-in and uses the answer
supplied by the plug-in, the answer is clearly a consequence of the plug-in
query.

• 2.

In the numerical problems (except the handful, such as C.1, where
the answer can be directly Googled) the only way to arrive at the correct
answer is to carry out a correct calculation. In those problems where GPT
does not call the plug-ins, it is certainly possible
that the AIs are using some other calculation than the one that
it describes, but there is no reason to suppose that.

In the multiple-choice and true-false problems of the “Calculation-Free”
test set, these considerations almost never apply — even when GPT-4 calls
the plug-in, it is not to compute the final answer — so there the
relation of the explanation to the answer is considerably more tenuous. Also,
the relation of explanations of things other than the numerical answer may
be questionable. For instance, if the LLM retrieves a particular formula
and gives a verbal explanation of why that formula is relevant, the
explanation may not be the actual reason that the LLM retrieved the formula.
The LLM may well have found the formula and explanation separately
by, in effect, matching the wording of the problem to similar problems
in the training
set, and retrieving the formula and explanation separately from those. For
instance, it would be possible for an LLM to retrieve a formula from one
source in the training set that discusses the problem and to retrieve
an unrelated explanation from another source discussing the same problem;
we did not observe any such behavior in our tests, but, as far as we know,
nothing in LLM technology makes that impossible.

References

Kurt Cobbe et al. (2021). Training verifiers to solve math word
problems.
https://arxiv.org/abs/2110.14168

Simon Frieder et al. (2023). Mathematical Capabilities of ChatGPT.
https://arxiv.org/abs/2301.13867

Dan Hendryks et al. (2021). Measuring mathematical problem solving with
the MATH dataset.
https://arxiv.org/abs/2103.03874

Pan Lu et al. (2022). Learn to explain: Multimodal reasoning via
thought chains for science question answering.
Advances in Neural Information Processing Systems, 35:2507–2521.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal (2018).
Can a suit of armor conduct electricity?
a new dataset for open book question answering.
Proceedings of the 2018 Conference on Empirical
Methods in Natural Language Processing (EMNLP).

Swaroop Mishra et al. 2022. Līla: A unified benchmark for
mathematical reasoning.
https://arxiv.org/abs/2210.17517

OpenAI (2023a). GPT4. March 14, 2023.
https://openai.com/research/gpt-4

OpenAI (2023b). GPT4 Technical Report. March 16, 2023.
https://arxiv.org/abs/2303.08774

OpenAI (2023c). ChatGPT plugins. March 23, 2023.
https://openai.com/blog/chatgpt-plugins

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP Models
really able to Solve Simple Math Word Problems?.
https://arxiv.org/abs/2103.07191

Aarohi Srivastava et al. 2022.
Beyond the imitation game: Quantifying and extrapolating
the capacities of language models.
https://arxiv.org/abs/2206.04615

Xiaoxuan Wang et al. (2023). SciBench: Evaluating college-level scientific
problem-solving abilities of large language models.
https://arxiv.org/abs/2307.10635

Steven Wolfram (2023). ChatGPT gets its “Wolfram superpowers”!. March 23,
2023.
https://writings.stephenwolfram.com/2023/03/chatgpt-gets-its-wolfram-superpowers/

Zicheng Ye et al. (2023).
Improving the Gilbert-Varshamov Bound by Graph Spectral Method.
CSIAM Transactions on Applied Mathematics, 4(1): 1–12
https://global-sci.com/article/90844/improving-the-gilbert-varshamov-bound-by-graph-spectral-method

Appendix A: The “Arbitrary Numerical” test set

The Matlab code included here was the code that we ourselves wrote to carry
out the calculations. It is included here
to facilitate checking the correctness of our answers.

• 1.

Object A is a spherical shell of gold, 1 meter in radius, 10 cm thick.
Object B is a spherical shell of copper, 10 cm in radius, 5 cm thick.
They are placed so that there is an 8 cm gap between them. What is the net
gravitational force they exert on each other?

Answer: 3.4486 dyne

Matlab

% By a theorem of Newton’s, the gravitational force exerted by spherical shell
% is the same as if all the mass were concentrated at the center.
GoldDensity = 19.3; % gm/cm^3 -- Wikipedia.
GoldVolume = (4/3) * pi * (100^3-90^3); %cm^3
GoldMass = GoldDensity * GoldVolume; % gm
CuDensity = 8.96 % gm/cm^3 --- Wikipedia
CuVolume = (4/3) * pi * (10^3-5^3); % cm^3
CuMass = CuDensity * CuVolume;
Distance = 100+10+8; %cm
GravConst = 6.674 * 10^(-8); % dyne cm^2/gm^2 -- Wikipedia
Force = GravConst * GoldMass * CuMass / Distance^2

• 2.

Let b, c, d be points at the center of Buenos Aires, Cincinnati, and Delhi,
respectively. Consider the plane P that contains b, c, and d (it cuts through
the earth). What is the area in square kilometers of the triangle b,c,d
lying in P?

Answer: 41,111,600 \mbox{km}^{2}.

Matlab

% Convert latitude and longitude to a unit 3D vector
function v=LatAndLongToUnit(LatDeg, LatMin, LatS, LongDeg, LongMin, LongS)
Theta = LatS*(LatDeg+(LatMin/60))*(pi/180);
Phi = LongS*(LongDeg+(LongMin/60))*(pi/180);
v = [cos(Theta)*cos(Phi), cos(Theta)*sin(Phi), sin(Theta)];
end

b=LatAndLongToUnit(34, 36, -1, 58, 22, -1)
c=LatAndLongToUnit(39, 6, 1, 84, 30, -1)
d=LatAndLongToUnit(28, 37, 1, 77, 14, 1)
bc=c-b
bd=d-b
cd=d-c
EarthRadius=6371; % The magnitude of the cross product
answer = norm(cross(bc,bd))*EarthRadius^2/2 % is the area of the parallelogram
x=norm(bc)
y=norm(bd)
z=norm(cd)
s=(x+y+z)/2
check = sqrt(s*(s-x)(s-y)(s-z))*EarthRadius^2 % Heron’s formula

• 3.

X is a regular tetrahedron of zinc with a side length of 2 moving at 100
m/sec. Y is a molecule of NaCl that is moving with speed v. X and Y have
equal kinetic energy. What is {}1-(v/c), where c is the speed of light?

Answer: {}8.41\cdot 10^{-21}

Matlab:

ZincVolume=2^3/(6*sqrt(2)) % Wolfram Alpha
ZincDensity = 7.14 % gm/cm^3. Wikipedia
ZincMass = ZincVolume*ZincDensity
VZinc = 100/(3*10^8) % Speed of Zinc in units where C=1
ZincKE = ZincMass*VZinc^2
NaClMass = 9.7*10^(-23) % gm. Wolfram Alpha
LorenzFactor = ZincKE/NaClMass
Q = 1/LorenzFactor^2 % 1-(v/c)^2
Answer = Q/2

Note: The above was the question, answer, and Matlab code that we published in
August 2023. We observed in September 2024 that the question omitted the measure of
length in describing the zinc prism, and that in the fifth line of our Matlab code,
the right side should be divided by 2. The correct answer to this question is
{}3.364\cdot 10^{-20}

• 4.

Viewed from Vega, what is the angle between Sirius and the Sun?

Answer: 0.0972 radians

Matlab:

AngleSEV = 2.755 % radians. Wolfram Alpha
DistES = 8.597 % light years. Wolfram Alpha
DistSV = 33.41 % light years. Wolfram Alpha
AngleEVS = asin(sin(AngleSEV)*DistES/DistSV)

• 5.

You have an empty cylindrical open container whose inside has a diameter of
8 centimeters and a height of 20 centimeters. and a pitcher with 200 ccs of water.
You first empty the pitcher into the cylinder, then
put a solid rock cube, 4 cm on a side, into the container so that it is
sitting flush against the
bottom of the container. What is the height of the water in the container?

Answer: 5.2521 cm

Matlab:

AreaOfCyl = pi*4^2
AreaAtBottom = AreaOfCyl-4^2
VolumeAtBottom = 4*AreaAtBottom
VolumeAtTop = 200 - VolumeAtBottom
HeightAtTop = VolumeAtTop/AreaOfCyl
Height = 4+HeightAtTop

• 6.

There are six small spherical masses placed at various positions in
space. Measured in a coordinate system with a 1 meter unit length:
A has mass 2 kg and is at \langle 2,0,0\rangle;
B has mass 3 kg and is at \langle 0,2,0\rangle;
C has mass 1 kg and is at \langle 0,0,2\rangle;
D has mass 4 kg and is at \langle 1,1,1\rangle;
E has mass 5 kg and is at \langle 1,0,1\rangle;
F has mass 2 kg and is at \langle 0,1,1\rangle.
What is the instantaneous acceleration of F under the gravitational attraction
of A,B,C,D,E? Express your answer as a vector.

Answer: \langle 0.403,-0.080,-0.056\rangle*10^{-9}\mbox{ m/sec}^{2}

Matlab:

% Gravitational force exerted on mass ma at point pa by mass
% mb at point pb, taking the gravitational constant to be 1.

function f = GravForce(ma,pa,mb,pb)
v = pb-pa;
r = norm(v);
u = v/norm(v);
f = mambu/(r^2)
end

g1 = GravForce(2,[0,1,1],2,[2,0,0]) + ...
GravForce(2,[0,1,1],3,[0,2,0]) + ...
GravForce(2,[0,1,1],1,[0,0,2]) + ...
GravForce(2,[0,1,1],4,[1,1,1]) + ...
GravForce(2,[0,1,1],5,[1,0,1])
GravConst = 6.67410^(-11) % Nkg^2/m^2 -- Wikipedia
acc = g1*GravConst/2

• 7.

How many total eclipses of the moon were there between Jules Verne’s
death and Neil Armstrong’s moon landing? An exact integer value is required.

Answer: 52.

Jules Verne died March 24, 1905.
Neil Armstrong landed on the moon July 20, 1969
There were 52 total lunar eclipses in between:
02-09-06, 08-04-06, 06-04-09, 11-27-09, 05-24-10, 11-17-10,
03-22-13, 09-15-13, 01-08-17, 07-04-17, 12-28-17, 05-03-20,
10-27-20, 04-22-21, 02-20-24, 08-14-24, 06-15-27, 12-08-27,
06-03-28, 11-27-28, 04-02-31, 09-26-31, 01-19-35, 07-16-35,
01-08-36, 05-14-38, 11-07-38, 05-03-39, 03-03-42, 08-26-42,
12-19-45, 06-14-46, 12-08-46, 04-13-49, 10-07-49, 04-02-50
09-26-50, 01-29-53, 07-25-53, 01-19-54, 11-18-56, 05-13-57,
11-07-57, 03-13-60, 09-05-60, 12-30-63, 06-25-64, 12-19-64,
04-24-67, 10-18-67, 04-13-68, 10-06-68
Source: Wikipedia article, “List of lunar eclipses in the twentieth century.”

• 8.

A quantity of chlorine gas is in a right prism whose base is a triangle with
sides 5 cm, 7 cm, and 4 cm and whose altitude is 8 cm. The temperature is
the freezing point of mercury, and the pressure is 2 atmospheres. What is the
mass of the chlorine?

Answer: 0.5781 gm

Matlab:

s=(5+7+4)/2
Area = sqrt(s*(s-5)(s-7)(s-4)) % Heron’s formula
VolumeCC = Area * 8
Liters = VolumeCC / 1000
Temp = 273.15 -38.83 % Wikipedia
IdealGasConst = 0.082057 % LatmK^{-1}*mol^{-1}. Wikipedia
Moles = Liters2/(TempIdealGasConst)
MassPerMoleCl2 = 70.9 % gm. Wolfram Alpha.
Mass = Moles * MassPerMoleCl2

• 9.

A train has two whistles, one at middle C and one at the F above middle C.
The train is driving past a station without stopping or changing speed.
On approaching the station, it blows
the low whistle; when it passes the station, it switches to the high whistle.
However, to the people standing at the station, it sounds like the whistle
dropped by a whole tone. How fast is the train moving?

Answer: 68.4 m/s

Explanation:
A half-tone in music is a factor of {}2^{1/12}. The interval
from C to F is 5 half-tones. So the effect is that a note as the train approaches is dropped by 7 half tones or a factor of
{}2^{7/12}.
Let s be the speed of sound, then (s+v)/(s-v)=2^{7/12}.
Solving for v we get

v=s\cdot\frac{2^{7/12}-1}{2^{7/12}+1}

The speed of sound is 343 m/s so the train
is moving at 68.4 m/s. (One can work this out by hand to an accuracy of 1% using
the fact that a fifth in music – 7 half tones – is 1.5 in the natural musical
scale.)

• 10.

A physical process generates photons whose energies follow a random distribution of the following form: For positive energy e, the probability
density at e is proportional to the value of e in a Gaussian distribution
with mean 2 Ev and standard deviation 0.01 Ev. The probability of a negative value is zero. What is the expected value of the wavelength of a
photon produced by this process? (Give the mathematical answer, assuming that the above description is exact, and assuming the standard relation between energy and wavelength in a photo. The answer is not physically plausible.)

Answer: Infinite.

Explanation: The wavelength of a photon with energy e is hc/e. The
mean wavelength of a photon in the beam is therefore

\int_{e=0}^{\infty}N_{2,0.01}(e)\cdot\frac{hc}{e}\>de

In the limit as e\rightarrow 0^{+}, the quantity N_{2,0.01}(e), though
extremely tiny, is still a positive quantity, so the integral diverges
at 0.

• 11.

The wavelengths of the photons in a beam of light are uniformly
distributed across the range of visible (to humans) light.
What is the mean energy of a photon in electron volts?

Answer: 2.3688 eV.

Explanation: The range of visible light is around 3800 to 7000 Angstroms.
A photon with wavelength λ has energy hc/\lambda. Thus, if λ
is uniformly distributed between 3800 and 7000, the mean energy is

\int_{\lambda=3800}^{7000}(1/3200)(hc/\lambda)\>d\lambda=(hc/3200)\log_{e}\lambda|^{7000}_{3800}

Matlab:

Planck = 4.136 * 10^(-15) % eV*sec
SpeedOfLight = 3*10^8 % m/sec
Meter2Ang = 10^(10)
SpeedOfLight * Meter2Ang * Planck / 3200 * log(7000/3800)

• 12.

A pendulum is hanging on a 2 meter cord attached to the ceiling 3
meters above the floor. It is brought to a position 25 degrees from the
vertical and released. It swings past the bottom and the cord is cut when it is
10 degrees from the vertical on the far side. Then the bob flies through the
air and hits the ground. What is the distance from the point where the bob
is released to the point where it hit the ground?

Answer: 2.3606 meters

% Take the point of attachment of the pendulum as the origin
g = 9.8 % Acceleration of terrestrial gravity in m/sec^2
d2r = pi/180 % Degrees to radians
s25 = sin(25*d2r) % Handy abbreviations
c25 = cos(25*d2r)
s10 = sin(10*d2r)
c10 = cos(10*d2r)
pr = [-2 * s25, -2*c25] % release point
pc = [2 * s10, -2*c10] % cut point
hd = 2 * (c10 - c25) % height difference between the cutpont and the release point.
s = sqrt(2ghd) % speed of the bob at release time
vc = s * [c10,s10]; % velocity of bob when cut
tca = vc(2)/g % time between cut and apex of flight
hca = (g*tca^2)/2 % height difference from cut point to apex
ha = -2*c10 + hca % height of apex of flight
haf = 3 + ha % height difference between apex and floor
taf = sqrt(2*haf/g) % time from apex to floor
pf = [pc(1) + (tca+taf)*vc(1),-3]
answer = norm(pf-pr)

• 13.

An spherical asteroid 500 km in diameter
travels in an essentially perfect circular orbit of radius
2.4 astronomical units. On January 4, 2023, the earth was at perihelion, and
as it happens, the asteroid was in perfect opposition. What was the solid angle
of the asteroid in the sky as seen by an earth observer?

Answer: {}4.35\cdot 10^{-12}

Matlab:

perihelion = 1.471 * 10^8 % km. Wikipedia
au = 1.498 * 10^8 % km. Wikipedia
asteroid = 2.4*au
distance = asteroid - perihelion
radiusAngle = 250/distance
solidAngle = pi*radiusAngle^2

• 14.

Two twin stars, one of 5 solar masses, the other of 10 solar masses,
orbit each other in circular orbits in a period of 4 earth years. How far apart
are they, in astronomical units?

Answer: 6.21 Au.

Explanation:
Let L be the distance between them in AUs. They each rotate around in a circle
centered at their center of gravity,
which is {}2L/3 from the small star and L/3 from the large star.
Let M=5 be the mass of the smaller star in solar masses.
Let Q be the ratio of the centripetal acceleration of the
large star to the centripetal
acceleration of the earth in its orbit.
Comparing the gravitational acceleration, Q=M/L^{2}=5/L^{2}
Comparing the centripetal acceleration, the star is moving on a path of
radius L/3 Au at an
angular velocity 1/4 of earth. Hence Q=L/48.
So L^{3}=5\cdot 48, so L=6.21 Au.

Matlab: Alternative solution using a generalized form of Kepler’s third
law:
a^{3}/T^{2}=G(m_{1}+m_{2})/4\pi^{2}.

Year = 3.154*10^7 %sec
Grav = 6.674310^(-11) % Nm^2/kg^2
SolarMass = 1.99*10^(30) % kg
M1 = 10*SolarMass
M2 = 5*SolarMass
Period = 4*Year
DistInMeter = (Grav*(M1+M2)Period^2/(4pi^2))^(1/3)
DistInAU = DistInMeter/(1.496*10^11)

• 15.

Draw a circle, on the earth’s surface, going through Cairo, Peking, and
Moscow. Let S be the area of the part of the earth’s surface inside the circle
and let P be the area of the circle in the plane of the circle. What is S/P?.

Answer: 1.108.

Matlab:

% Since we are computing a ratio, we take the earth’s radius to be 1
c = LatAndLongToUnit(30,2,1,31,14,1) % Unit vector corresponding to Cairo
p = LatAndLongToUnit(39,54,1, 116,24,1) % Peking
m = LatAndLongToUnit(55,45,1,37,37,1) % Moscow
a = zeros(3); % coefficient matrix
q = zeros(3,1); % constant terms
% Equation of the perpendicular bisector plane of line cp is dot(x,p-c) = 0
% (Note that the origin is always in the plane.)
a(1,:) = p-c
% Likewise perpendicular bisector of cm
a(2,:) = m-c
a(3,:) = cross(p-c,m-c) % normal to plane of three cities
q(3,1) = dot(c,a(3,:))
f = a\q % circumcenter is intersection of these 3 planes
f=f’;
r = norm(f-c) % radius of circumcenter
norm(m-f) % check for correctness
norm(p-f)
PlanarArea = pi*r^2 % area of planar circle
SphereArea = 2pi(1-norm(f)) % Formula from Wikipedia "Spherical sector"
Answer = SphereArea/PlanarArea

• 16.

Assume that the probability of having any particular isotope of a chemical
follows their frequency on earth. What is the probability that a randomly
constructed molecule of glucose will have
4 atoms of {}^{12}\mbox{C},
2 atoms of {}^{13}\mbox{C},
11 atoms of {}^{1}\mbox{H},
1 atom of {}^{2}\mbox{H},
3 atoms of {}^{18}\mbox{O},
3 atoms of {}^{16}\mbox{O}.

Answer: {}4.797\cdot 10^{-13}

Matlab:

nchoosek(6,4) * .0106^2 *0.989^4 * ...
nchoosek(12,1) * .99985^11 *0.000145 * ...
nchoosek(6,3) * 0.998^3 * 0.00205^3
Answer: 4.797 * 10^-13

• 17.

Two {}^{31}K phosphorus nuclei, with no electrons, are isolated in
space, with coordinates \langle 0,0,0\rangle and \langle 10,10,10\rangle in a coordinate
system whose unit length is 1 Angstrom. What is the instantaneous acceleration
(a vector with unit length of Angstrom/sec2)
of the nucleus at the origin due to the electrostatic force?

Answer: \langle-1.9420\cdot 10^{27},-1.9420\cdot 10^{27},-1.9420\cdot 10^{27}\rangle

Matlab

Mass = 5.1433*10^(-26) % kg
Charge = 151.60210^(-19) % coulomb
CoulombConst = 8.988 * 10^9 % Newton * m^{2}/C^{2}
Angstrom2Meter = 10^(-10)
Distance = 10*sqrt(3)*Angstrom2Meter
Force = CoulombConst*Charge^(2)/Distance^(2) % Newtons
Acceleration = -Force/(Mass*Angstrom2Meter) * [1,1,1]/sqrt(3)

Note: The above was the question, answer, and Matlab code that we published in
August 2023. We noticed in September 2024 that the question was mistated; the
symbol for phosphorus is P, not K. Our calculation was right on the assumption that
phosphorus was intended.

• 18.

An irregular (house-shaped) pentagon has vertices numbered 1 through 5 in
order. The pentagon has right angles at vertices 1, 3, and 4, and 135-degree
angles at 2 and 5. Side 2-3 and 4-5 have length 1 and side 3-4 has length 2.
The pentagon is placed on a planar coordinate system so that the numbering of
the vertices is in clockwise order, vertex 3 is at the origin, and vertex 5
is on the positive y-axis. What are the coordinates of vertex 1?

Answer: \langle 1.5269,2.3446\rangle

Matlab:

% Problem 18
% Consider the pentagon in a standard orientation where vertex 3 is at the origin and
% vertex 4 is on the negative x-axis. Compute the rotation necessary to place 5 above
% 3, and then apply that rotation to 1.

theta = atan2(1,-2)-(pi/2)
standard1 = [-sqrt(2);1+sqrt(2)]
answer = [cos(theta), sin(theta); -sin(theta), cos(theta)]*standard1
standard5 = [-2;1]
check = [cos(theta), sin(theta); -sin(theta), cos(theta)]*standard5 % check

• 19.

Consider a cube with unit length sides, where the vertices of one
face are numbered A..D in counterclockwise order, as viewed from the center
of the cube; the vertices of the opposite face are named E to H; and there are
edges AE, BF, CG, and DH.
Rotate the cube so that vertex A is at the origin,
vertex G is on the positive z axis, and vertex B is in the x-z plane, with
positive x coordinate.
What are the coordinates of vertex E?

Answer: \langle-0.4082,-0.7071,.5774\rangle=\langle-\sqrt{1/6},-\sqrt{1/2},\sqrt{1/3}\rangle

Explanation:
Consider the cube in a standard position where \vec{a}=\vec{0},
\vec{b}=\hat{i}, \vec{d}=\hat{j}. \vec{e}=\hat{k}; thus
\vec{g}=\langle 1,,1,1\rangle. Since \vec{a} is at the origin after the rotation,
this is a pure rotation around the origin; it is therefore equivalent to
multiplication by an orthogonal matrix m with determinant 1.
The dot product m(2,:) with any vector v is the y-coordinate of the
rotated position of v. In particular since after the rotation both \vec{b}
and \vec{g} have y-coordinate zero, m(2,:) must be orthogonal to both;
hence, it is their cross-product, normalized. Since the x coordinate of
the rotated place of g is 0, and since m is orthogonal, m(1,;) is
orthogonal to both \vec{g} and m(2,:); hence, it is the normalized
form of their cross product. Since m is orthogonal, m(3,:) is the
cross product of m(1,:) and m(2,:). All that
remains is to make sure the signs are OK, to check, and to compute
m\cdot\vec{e}.

Matlab

% Problem 19

b = [1,0,0]
g = [1,1,1]
e = [0,0,1]
m = zeros(3);
u = cross(b,g) % both b and g get mapped to a vector with y component = 0
m(2,:) = u / norm(u)
v = cross(m(2,:),g)
m(1,:) = sign(dot(v,b))*v/norm(v)
m(3,:) = cross(m(1,:),m(2,:))
if (dot(m(3,:),g) < 0)
m(2,:) = -m(2,:)
m(3,:) = -m(3,:)
end
m*m’ %Check: rotation matrix?
det(m)
m*b’ %Check: b mapped into xz plane with positive x?
m*g’ %Check: g mapped into positive z axis?
answer = m*e’

• 20.

Joe and Jim each have a bank account which they started on January 1,

2000. Joe started his account with with $1000; Jim started his with $ 900.
      Joe’s account pays 5% every December 31, and he
      adds an additional $500 every January 1. Jim’s account pays 10% every
      December 31. When will Jim’s account have more money in it than Joe’s?

Answer: 2053. (More precisely, December 31, 2052).

Matlab:

joe = 1000
jim = 900
count = 0
while (jim <= joe)
count = count+1
jim = jim*1.10
joe = joe*1.05 + 500
end
2000+count

• 21.

Two points p and q are independently randomly
chosen following
a uniform distribution in a three-dimensional
sphere of radius {}10^{7}. With a relative error of
less than 1%, give an estimate with an error no greater than 1%
that the Euclidean distance from
p to q is less than 1?

Answer: {}10^{-21}.

Explanation: For any point p not close to the boundary of the
large sphere, q will be within distance 1 of p if and only if
it is inside the sphere radius 1 around p. That breaks down if p
is within 1 of the boundary of the large sphere, but the probability of that
is tiny (around {}4\pi\cdot 10^{-7}). So the probability that q is inside
the small sphere given that it is in the large sphere is the ratio of the volumes
of the spheres = {}10^{-21}.

• 22.

A point p is chosen at random within the 100-dimensional box
{\bf B}=[0,100]^{100} following a uniform distribution.
What is the probability
that the Euclidean distance from p to the boundary of \bf B
is less than 1?

Answer: {}1-0.98^{100}\approx 1-1/e^{2}=0.8647

Explanation: The distance from p to the boundary is
\min_{i=1..k}\min(p_{k},100-p_{k}) That is, the distance is more than 1
if and only if {\bf p}\in[1,99]^{100}. So the probability that the distance
is less than one is (100^{100}-98^{100})/100^{100}.

• 23.

A point p is chosen at random within the 100-dimensional box
{\bf B}=[0,1]^{100} following a uniform distribution. What is the mean
value of the Euclidean distance from p to the boundary of B?

Answer: 1/202.

Explanation: Pick one particular face of the hypercube; for convenience
the lower face in the x-dimension {\bf F}=\{0\}\times[0,100]^{99}.
Let {\bf c}=\langle 0.5,0.5,\ldots 0.5\rangle
be the center of B. Let P
be the hyperpyramid consisting of the convex hull of F \cup { c
}. It is obvious, and easy to show, that the points in B whose
closest boundary point is F are exactly those in P and that the
distance of a point in P to F is just its x-coordinate. Moreover,
since B can be partitions into such pyramids, one for each face of
B and they all have identical distributions of distances to the boundary,
the mean distance to the boundary is the same in P. Therefore the mean
distance for a point in G to the boundary of B
is the same as mean distance for the distance from a point in P
to F which is just the mean x-coordinate of a point in P.

The rest is just calculus. The cross-section of P at x-coordinate x
is the box [x,1-x]^{99} of 99-d volume (1-2x)^{99}. We could integrate
this to compute the volume of P but it’s simpler to note that
there is one such pyramid for each face of B; there are 200 such faces;
and their union of the pyramids is all of B which has volume 1. So the
volume of P is 1/200. So the probability density of a uniform choice
in P as a function of x is {}200(1-2x)^{99}.

The expected value of x in P is therefore

\int_{x=0}^{0.5}200(1-2x)^{99}xdx=200\int_{u=0}^{1}u^{99})\cdot((1-u)/2)\cdot(du/2)=(200/4)\int_{u=0}^{1}u^{99}-u^{100}du=

(200/4)\cdot(u^{100}/100-u^{101}/101)|^{1}_{0}=1/202.

• 24.

A pound of graphite is inside a closed cylindrical container with a radius
of 50 cm and a height of 200 cm, filled with air at room
temperature ({}22^{\circ} C) and atmospheric pressure. The graphite
reacts with the oxygen, producing equal masses of carbon monoxide and
carbon dioxide until one of the reactant chemicals is exhausted. How
many moles of each chemical is in the container at the end?

Answer: No oxygen, 11.86 moles of carbon monoxide, 7.61 moles of
carbon dioxide, 18.2 moles of carbon.

Matlab:

CAtomMass = 12.011
OAtomMass = 16
COMolecMass = CAtomMass+OAtomMass
CO2MolecMass = CAtomMass+2*OAtomMass
MolesOfCarbon = 453.6/CAtomMass % Wikipedia
Temp = 273.15 + 22 % Wikipedia
Liters = pi20050^2/1000 % Volume in liters
IdealGasConst = 0.082057 % latmK^{-1}*mol^{-1}. Wikipedia
MolesOfAir = Liters/(Temp*IdealGasConst)
MolesOfO2 = MolesOfAir * .2095
MoleFracOfCO = CO2MolecMass/(COMolecMass+CO2MolecMass)% Molar fraction of CO in product
A = 2*MolesOfO2/(2-MoleFracOfCO)
MolesOfCO = MoleFracOfCO * A
MolesOfCO2 = (1-MoleFracOfCO) * A
MolesOfCOCOMolecMass - MolesOfCO2CO2MolecMass %check
MolesOfCO + 2MolesOfCO2 - 2MolesOfO2 %check
EndMolesOfCarbon = MolesOfCarbon - (MolesOfCO+MolesOfCO2)

• 25.

Suppose that you have a gram of pure radium 223. What is the mass of
the helium that that will generate via fission in a week?
Answer: 0.0248 gm

Explanation: Radium 223 undergoes a series of nuclear reactions:
Radium 233 → Radon 219 + alpha
Radon 219 → Polonium + alpha
Polonium 215 → Lead 211 + alpha
Lead 211 → Bismuth 211 + beta
Bismuth 211 → Thallium 207 + alpha
Thallium 207 → Lead 207 + beta
Thus each atom of radium generates 4 alpha particles.

The first reaction has a half-life of 11.43 days; the others have half-lifes ranging
from fractions of a second to a few minutes.

Matlab:

radiumMass = 223
heliumMass = 4
halfLife = 11.43 % Half life of radium 223 in days
f = (1/2)^(7/halfLife)
answer = (1-f)4heliumMass/radiumMass

• 26.

A small school has 100 students and three extramural activities: the basketball
team, the chess club, and the drama society. There are 60 students in the band,
48 in the chess club, and 55 in the drama club. There are 18 students who
do both band and chess, 28 who do both band and drama, 31 who do both chess
and drama, and 8 students who are in all three groups.
What is the probability that a student is not in the band given
that they are either in the drama club or the chess club but not both?

Answer: 11/41

Explanation: It is easily determined from the above information that
BCD=8.
BC\bar{D}=10
B\bar{C}D=20.
B\bar{C}\bar{D}=22.
\bar{B}CD=23.
\bar{B}C\bar{D}=7.
\bar{B}\bar{C}D=4.
\bar{B}\bar{C}\bar{D}=6.
The above answer is immediate.

• 27.

You draw 20 cards from a deck of cards labelled 1..60. To your surprise,
the cards can be arranged as an arithmetic sequence. What is the probability
that the successive difference is 3?
Give an exact answer as a rational number.

Answer: 3/66 = 1/22.

Explanation:
There are 41 sequences of difference 1 (starting with every
value from 1 to 41); 22 sequences with difference 2 (starting from 1 to 22);
and 3 with difference 3 (starting points 1, 2, and 3).

• 28.

You draw 5 cards from a deck of cards labelled 1 … 200. To your
surprise the cards can be arranged as a geometric series. What is the
probability that the smallest number is less than or equal to 7?
(Note that the ratio need not be an integer.)

Answer: 9/16

Explanation:
The possible increasing ratios are 2, 3, and 3/2.

If the ratio is 2, the sequence has the form
a, 2a, 4a, 8a, 16a. a can be any integer from 1 to 12.
In 7 of these, a ≤ 7.

If the ratio is 3, the sequence has the form
a, 3a, 9a, 27a, 81a. a is either 1 or 2.
In both of these, a ≤ 7.

If the ratio is 3/2, the sequence has the form
16a, 24a, 36a, 54a, 81a, and a can be either 1 or 2. In both of them
the smallest number is greater than 7.

So the probability is 9/16.

• 29.

In this question and the following three,
assume that the satellite is moving in a closed orbit around the Earth
and that the only influence on the satellite’s motion is the
Earth’s gravity. Assume that the Earth is a perfect sphere.
Ignore the revolution of the Earth around the sun,
but do not ignore the rotation of the Earth around its axis.

A satellite in a circular geosynchronous orbit remains directly above the
point on the earth’s equator {}0^{\circ} N, {}50^{\circ} E. What is the
straight line distance from the satellite to Houston?

Answer: 46,934 km.

Matlab:

% Method 1: Compute the positions of Houston and the satellite
% as 3 dimensional vectors
r = 6378.1 % km. Earth’s radius
q = 35786 % height of geosynchronous orbit over earth’s surface
rs = r+q % distance from center of earth to satellite
deg2rad = pi/180; % degrees to radians
hlat = 29.75*deg2rad % Houston latitude and longitude
hlong = -95.38*deg2rad
uh = [cos(hlat)*cos(hlong), cos(hlat)*sin(hlong),sin(hlat)] % unit vector Houston
h = r*uh % Houston location as vector
slong = 50*deg2rad
us = [cos(slong),sin(slong),0]
s = rs*us % Satellite location as vector
answer1 = norm(s-h)

% Method 2 Apply law of cosines to triangle Houston - center of earth- satellite
cosTheta = dot(us,uh) % cosine of angle at center of earth
sqrt(r^2 + rs^2-2cosThetar*rs)

• 30.

A satellite in a circular geosynchronous orbit passes directly above the
North and South Poles. When it crosses the North Pole, its velocity is in the
plane of the {}0^{\circ} circle of longitude. At what longitude
does it pass directly above a point in the Tropic of Cancer?
(An answer will be marked correct if it is within 3 degrees of the correct
answer.)

Answer: {}66.56^{\circ} W.

Explanation: The period of a geosychronous orbit is equal to a sidereal
day. In that time, the satellite executes a {}360^{\circ} revolution and the
earth executes a {}360^{\circ} rotation, so their angular velocities are equal.
So the longitude is equal to the difference in latitude between the North Pole
and the Tropic of Cancer. Since the Tropic of Cancer is at 23.44 degrees, the
difference is 66.56 degrees.

• 31.

A satellite orbits the earth in a circular orbit.
It completes an orbit every 14 hours 40 minutes. How high is
it above the Earth’s surface?

Answer: 24,042 km

Matlab:

% By Kepler’s 3rd law, R^3 is proportional to T^{2}, so R is proportional to T^(2/3)
r = 6378.1 % km. Earth’s radius
q = r+35786 % km. distance of geosynchronous orbit from center of earth
e = 23+(56/60) % hours. Period of geosynchronous orbit.
z = q*((14+(40/60))/e)^(2/3) % Distance from satellite to center of earth
answer = z-r

• 32.

A satellite orbits the earth in a circular orbit. It passes directly over the
North and South poles and completes an orbit every 14 hours 40 minutes.
On one orbit going southward it was directly above the earth location
{}40^{\circ} N,
{}10^{\circ} W at 1:00 PM EST. At what time will it next cross the
plane that contains
the circle of latitude {}40^{\circ} N, and what will be its longitude?
(An answer will be marked correct if the time is within 5 minutes of the
correct answer, and the longitude is within 3 degrees.)

Answer: Time: 2:19 PM. Longitude: 29.77 degrees W (= {}20^{\circ}\circ 40^{\prime} W)

Matlab:

% Let c be the center of the earth.
% Let P be the plane containing the circle of 40 degrees latitude.
% Let s be the position of the satellite when it crosses P
% Let a be the point where the line from c to the North Pole crosses P.
% Let r be the radius of the earth and let h be the height of the satellite

% cas is a right triangle. |ca| = r*sin(40 degrees). |cs| = r+h
% So the angle acs = arcsin(|ca|/|cs|).
% The angle traversed since 1 PM is 40-(90-acs)
% The rest of the calculation is as in problem 29

r = 6378.1 % km. Earth’s radius
h = 24042 % height of satellite over Earth’s surface. Problem 30
w = 14+(40/60) % period of satellite
d = 23+(56/60) % sidereal day
deg2rad = pi/180
ca = rsin(40deg2rad)
cs = r+24042 % Problem 30
acs = asin(ca/cs)
at = 40-(acs/deg2rad) % angle traversed in degrees
lat = 10 + (w/d)*at % latitude
t = 1 + w*(at/360) % time

Appendix B: The “Calculation-Free” test set

Eclipse problems

• 1.

An astronaut is standing in the Sea of Tranquility during what on
earth is called a total lunar eclipse. They are looking in the
direction of the earth. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
B. The night side of the earth, occluding the sun.

• 2.

An astronaut is standing in the Sea of Tranquility during what on
earth is called a total lunar eclipse. They are looking in the
direction of the sun. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
B. The night side of the earth, occluding the sun.

• 3.

An astronaut is standing in the Sea of Tranquility during what on
earth is called a total solar eclipse. They are looking in the
direction of the earth. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
F. The day side of the earth, with a small circular shadow moving
quickly over it.

• 4.

An astronaut is standing in the Sea of Tranquility during what on
earth is called a total solar eclipse. They are looking in the
direction of the sun. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
A. The surface of the moon, illuminated by earth light.

• 5.

An astronaut is standing on the so-called “dark side of the moon”,
during what on
earth is called a total lunar eclipse. They are looking in the
direction of the earth. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
C. The surface of the moon, illuminated only by starlight.

• 6.

An astronaut is standing on the so-called “dark side of the moon”,
during what on
earth is called a total lunar eclipse. They are looking in the
direction of the sun. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
C. The surface of the moon, illuminated only by starlight.

• 7.

An astronaut is standing on the so-called “dark side of the moon”,
during what on
earth is called a total solar eclipse. They are looking in the
direction of the earth. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
D. The surface of the moon, illuminated by the sun.

• 8.

An astronaut is standing on the so-called “dark side of the moon”
during what on
earth is called a total solar eclipse. They are looking in the
direction of the sun. What they see is:

A.

The surface of the moon, illuminated by earth light.

• B.

The night side of the earth, occluding the sun.

• C.

The surface of the moon, illuminated only by starlight.

• D.

The surface of the moon, illuminated by the sun.

• E.

The sun.

• F.

The day side of the earth, with a small circular shadow moving
quickly over it.

• G.

The night side of the earth. The sun is somewhere else entirely.

• H.

A starry sky. Neither the sun, the earth, or the surface of the moon
is in the field of view.

Answer:
E. The sun.

Distance combination problems

• 9.

Joe says that he lives 10 miles from the Atlantic, that Beth lives
10 miles from the Atlantic, and that he and Beth live 3000 miles apart.
Is it possible that Joe is telling the truth? Answer Yes

• 10.

Joe says that he lives 10 miles from New York City, that Beth lives
10 miles from New York City, and that he and Beth live 3000 miles apart.
Is it possible that Joe is telling the truth? Answer No.

• 11.

Joe says that he lives 10 miles from Lake Michigan, that Beth lives
10 miles from Lake Michigan, and that he and Beth live 100 miles apart.
Is it possible that Joe is telling the truth? Answer Yes

• 12.

Joe says that he lives 10 miles from Walden Pond, that Beth lives
10 miles from Walden Pond, and that he and Beth live 100 miles apart.
Is it possible that Joe is telling the truth? Answer No,

• 13.

Joe says that he lives 100 miles from Walden Pond, that Beth lives
100 miles from Lake Michigan, and that he and Beth live 10 miles apart.
Is it possible that Joe is telling the truth? Answer No.

• 14.

Joe says that he lives 1000 miles from Walden Pond, that Beth lives
1000 miles from Lake Michigan, and that he and Beth live 10 miles apart.
Is it possible that Joe is telling the truth? Answer Yes

• 15.

Joe says that he lives 10 miles from Lake Huron, that Beth lives
10 miles from Lake Michigan, and that he and Beth live 10 miles apart.
Is it possible that Joe is telling the truth? Answer Yes.

• 16.

Is there a point in the Nile River that is exactly 827 miles from the
Al-Azhar Mosque in Cairo? Answer Yes.

• 17.

Is there a point in the Nile River that is exactly 8562 miles from the
Al-Azhar Mosque in Cairo? Answer No.

• 18.

Is there a point x in the Danube River and a point y in the Rhine River
that are exactly 12 miles apart? Answer No.

• 19.

Is there a point x in the Danube River and a point y in the Rhine River
that are exactly 429 miles apart? Answer Yes.

• 20.

Is there a point x in the Danube River and a point y in the Rhine River
that are exactly 8738 miles apart? Answer No.

Clockwise vs. counterclockwise

If you have a map that shows Chicago, New York City, and Atlanta, and
you draw a circle through the three of them,
then the sequence \langleChicago; New York; Atlanta\rangle is in clockwise order.
The sequence
\langleNew York; Chicago; Atlanta\rangle on the other hand, is in
counterclockwise order.

For each of the following, state whether it is in clockwise or counterclockwise
order:

• 21.

Caracas, Venezuela;   Amarillo, Texas;
Quebec, Quebec.
Answer: Clockwise.

• 22.

New Orleans, Louisiana;   Springfield, Illinois;
Jacksonville, Florida.
Answer: Clockwise.

• 23.

Kingston, Jamaica;   Fresno, California;
Albany, New York.
Answer: Clockwise.

• 24.

Indianapolis, Indiana;   Wichita, Kansas;
Montgomery, Alabama.
Answer: Counterclockwise.

• 25.

Denver, Colorado;   Houston, Texas;
Indianapolis, Indiana.
Answer: Counterclockwise.

• 26.

Seattle, Washington;   New York, New York;
Mazatlan, Mexico.
Answer: Clockwise.

• 27.

Spokane, Washington;   Mexico City, Mexico;
Virginia Beach, Virginia.
Answer: Counterclockwise.

• 28.

Sitka, Alaska;   Oakland, California;
Albany, New York.
Answer: Counterclockwise.

• 29.

Pierre, South Dakota;   Eastport, Maine;
Columbia, South Carolina.
Answer: Clockwise.

• 30.

Toronto, Ontario;   El Paso, Texas;
Key West, Florida.
Answer: Counterclockwise.

• 31.

Sitka, Alaska;   Omaha, Nebraska;
Long Beach, California.
Answer: Clockwise.

• 32.

Edmonton, Alberta;   New Orleans, Louisiana;
Washington DC.
Answer: Counterclockwise

Sorting problems

Sorting problems should be scored in terms of the number of pairs in order
divided by the total number of pairs. The order of answers has been randomized.

• 33.

Sort the items below in increasing order of mass:

A.

Pablo Picasso, when five years old.

• B.

The Great Sphinx

• C.

An atom of uranium.

• D.

The planet Mercury.

• E.

A Toyota Corolla

• F.

The polio virus.

• G.

A hamster.

• H.

Emily Dickinson, when twenty-three years old.

Answer: C, F, G, A, H, E, B, D.

• 34.

Sort the events below by duration:

A.

The lifetime of Marie Antoinette.

• B.

The Precambrian period.

• C.

The first performance of Beethoven’s seventh symphony.

• D.

Lincoln speaking the Gettysburg address.

• E.

The Hundred Years’ War.

• F.

The reign of Queen Victoria.

• G.

The existence of the species of passenger pigeons (ending with
the death of “Martha”).

• H.

The battle of Gettysburg.

• I.

The existence of legal slavery in what is now the United States.

• J.

The reign of Marie Antoinette.

• K.

The lifetime of Joan of Arc.

Answer: D, C, H, K, J, A, F, E, I, G, B

(Note: The durations of K, the lifetime of Joan of Arc, and J, the reign of
Marie Antoinette, are very close – about 19 years. Moreover, the starting
date of the former is not known precisely, and the ending date of the latter
is debatable. Either order
should be accepted as correct for these. This was included by misdesign.)

• 35.

Sort the events below by starting date:

A.

The lifetime of Marie Antoinette.

• B.

The Precambrian period.

• C.

The first performance of Beethoven’s seventh symphony.

• D.

Lincoln speaking the Gettysburg address.

• E.

The Hundred Years’ War.

• F.

The reign of Queen Victoria.

• G.

The existence of the species of passenger pigeons (ending with
the death of “Martha”).

• H.

The battle of Gettysburg.

• I.

The existence of legal slavery in what is now the United States.

• J.

The reign of Marie Antoinette.

• K.

The lifetime of Joan of Arc.

Answer: B, G, E, K, I, A, J, C, F, H, D

• 36.

Sort the events below by ending date:

A.

The lifetime of Marie Antoinette.

• B.

The Precambrian period.

• C.

The first performance of Beethoven’s seventh symphony.

• D.

Lincoln speaking the Gettysburg address.

• E.

The Hundred Years’ War.

• F.

The reign of Queen Victoria.

• G.

The existence of the species of passenger pigeons (ending with
the death of “Martha”).

• H.

The battle of Gettysburg.

• I.

The existence of legal slavery in what is now the United States.

• J.

The reign of Marie Antoinette.

• K.

The lifetime of Joan of Arc.

Answer: B, K, E, J, A, C, H, D, I, F, G.

• 37.

Sort these entities in increasing order of distance from the painting
“The Mona Lisa”.

A.

Omaha Beach

• B.

Phobos, the moon of Mars.

• C.

The Dome of the Rock

• D.

The Washington Monument

• E.

The Sistine Chapel

• F.

Leonardo da Vinci’s painting, “St. John the Baptist”.

• G.

The Tomb of Napoleon.

• H.

The Rosetta Stone

• I.

The Parthenon

• J.

Mount Fujiyama

• K.

The Great Barrier Reef.

Answer: F, G, A, H, E, I, C, D, J, K, B

• 38.

Sort these people by increasing order of the distance of their
birthplace from the birthplace of Frederic Chopin.

A.

Gandhi

• B.

Mao

• C.

Cleopatra

• D.

Kafka

• E.

Mohammed

• F.

Cate Blanchett

• G.

Napoleon

• H.

Marie Curie

Answer: H, D, G, E, A, B, F.

• 39.

Sort these people by increasing order of the distance of their birthplace
from the birthplace of Abraham Lincoln.

A.

Ruth Bader Ginsburg

• B.

The Duke of Wellington

• C.

Ulysses Grant

• D.

Vladimir Lenin

• E.

Jacindra Ardren.

• F.

Ho Chi Minh

• G.

Kamala Harris

• H.

Barack Obama

• I.

George Washington

Answer: C, I, A, G, B, H, D, E, F

Note: It is not at all obvious, without doing the calculation, that Abraham
Lincoln’s birthplace (Hodgenville, Ky.) is closer to the Duke of Wellington’s
(Dublin) or that it is closer to Jacindra Ardern’s birthplace (Hamilton,
New Zealand) than to Ho Chi Minh’s (Nghe An province, Vietnam). These
comparisons were included due to an error in design.

Satellites

In the following assume that the satellite is moving in a closed
orbit around the Earth
and that the only influence on the satellite’s motion is the
Earth’s gravity. Ignore the revolution of the Earth around the sun,
but do not ignore the rotation of the Earth around its axis. If there are two
or more satellites, assume that it is not possible for them to occupy the same
point in space simultaneously.

• 40.

Is it possible to have a satellite such that the point on the earth
underneath the satellite always has longitude {}40^{\circ} W?
Answer: Yes (a satellite in geostationary orbit
that is always over the earth
point on the equator at {}30^{\circ} S, {}40^{\circ} W).

• 41.

Is it possible to have a satellite such that the earth point underneath
the satellite always has latitude {}40^{\circ} S?
Answer: No.

• 42.

Is it possible to have a satellite such that the northernmost
earth point underneath the satellite has latitude {}40^{\circ} N and the
southernmost has latitude {}30^{\circ} S?
Answer: No.

• 43.

Assume that a satellite passes in its orbit over the North Pole. Which of
the following is true:

A.

It must pass over the South Pole.

• B.

It cannot pass over the South Pole.

• C.

It might or might not pass over the South Pole.

Answer: A

• 44.

Suppose that the orbit of a satellite, which is non-circular,
takes it over the North Pole, and
moreover, it is furthest from the earth when it is over the North Pole.
When will it be nearest to the earth?

A.

When it is over the equator.

• B.

When it is over the South Pole.

• C.

At some point after it has crossed the equator but before it has
passed over the South Pole.

• D.

None of the above are necessarily true.

Answer: B

• 45.

Can two earth satellites have their orbits in planes that are not
equal but are parallel?
Answer: No

• 46.

Can two earth satellites have their orbits in planes that are
orthogonal?
Answer: Yes

• 47.

Let C be the center of the earth.
Can there be two earth satellites X and Y
such that X, C, and Y always lie in a straight line, with C between X and Y?
Answer: Yes

• 48.

Let C be the center of the earth.
Can there be two earth satellites X and Y
such that C, X, and Y always lie in a straight line, with X between C and Y?
Answer: No.

• 49.

Let C be the center of the earth.
Can there be two earth satellites X and Y
such that the angle XCY is always a right angle?
Answer: Yes.

• 50.

Let C be the center of the earth.
Can there be two earth satellites X and Y
such that the angle CXY is always a right angle?
Answer: No.

• 51.

Let C be the center of the earth.
Can there be two earth satellites X and Y
such that the angle CXY is always 60 degrees?
Answer: Yes.

• 52.

Let C be the center of the earth.
Can there be three earth satellites X, Y, and Z such that C, X, Y, and Z
are always coplanar?
Answer: Yes.

• 53.

Let C be the center of the earth.
Can there be three earth satellites X, Y, and Z
such that the lines CX, CY, and CZ are always all pairwise orthogonal?
Answer: No.

Appendix C: The “Motivated Numerical” test set

• 1.

Question:
If the earth collapsed to a black hole, how big would the black hole
be? Please include all calculations.

Answer: 8.87 millimeters.

• 2.

If you fell into the black hole at the center of the Milky Way, how
long would you have before hitting the singularity? Please include all
calculations.

Answer: 66.7 seconds.

• 3.

Consider the first binary black hole system discovered by LIGO.
Roughly how close to that system would a person have to have been,
before they were killed by the gravitational waves? Please include all
calculations.

Answer: So close that you would be killed by the tidal forces rather than
by the gravitational waves.

• 4.

How far off from 10! is Stirling’s approximation? Please include all calculations.

Answer: 30,104.

• 5.

How high would an airplane have to be, before you could notice 10
degrees of the earth’s curvature when looking out the window?
Please include all calculations.

Answer: 98.4 km.

Matlab

r = 6378 % Radius of earth in kilometers
d = r/cos(10*pi/180) % distance from center of earth to airplane
answer = d-r %height above earth’s surface

• 6.

Approximately how much time would a commercial airliner save in going from New York
to Tel Aviv, if it could go in a straight line, through
a tunnel in the earth, at the same speed as usual? Please include all
calculations.

Answer: About 52 minutes, assuming a speed of 550 mph.

Matlab:

nyclat = (40+42/60+46/3600)*pi/180
nyclong = -(74+22/3600)*pi/180
talat = (32+8/60)*(pi/180)
talong = (34+78/60)*(pi/180)
earthRadius = 3959 % miles
nycDirection = [cos(nyclat)*sin(nyclong), cos(nyclat)*cos(nyclong), sin(nyclat)] % unit vector
taDirection = [cos(talat)*sin(talong), cos(talat)*cos(talong), sin(talat)]
tunnel = earthRadius*norm(taDirection-nycDirection)
flight = earthRadius*acos(dot(nycDirection,taDirection))
timeSaved = (flight-tunnel)/550 % assuming 550 mph.

• 7.

For what fraction of the lifetime of the universe has there been
life in it? Give upper and lower bounds.
Please include all calculations.

Answer: Between 25% and 99.93%.

Explanation: The age of the universe is taken to be 13.8 billion years.
It is generally accepted that life emerged on earth at least 3.5 billion years
ago, and it is speculated that it might have been possible for life to appear
in the universe about 10 years after the Big Bang.

• 8.

Approximately how long would it take to transmit an entire human
genome over a standard WiFi connection?
Please include all calculations.

Answer: 4 minutes.

Explanation: The human genome has about 3 billion base pairs = 6 billion
bits. Assuming a 25Mbs Wifi connection, that gives 240 seconds = 4 minutes.

• 9.

Approximately how large would an asteroid have to be, in diameter
(assume it’s approximately spherical), before an Olympic high jumper
could no longer reach escape velocity by jumping off it?
Please include all calculations.

Answer About 10 km.

Explanation: The kinetic energy needed for escape velocity from the
surface of a planet is inversely proportional to the radius and proportional
to the mass of the planet, which in turn is proportional to the
density and radius cubed.
So over planets of constant density energy is proportional to the radius
squared and density.
So the velocity is proportional to the radius and square root of the density.
So the radius as a proportional to the velocity and inversely proportional
to the square root of the density.

The world record for high jump is 2.45 meters, set by Javier Sotomayor in 1993.
The jumper’s center of mass starts about half way up his body, around .9 m.
(Sotomayor is 193 cm tall.) At the apex of the jump, it is perhaps .3m above
the bar. So the jump raises the center of mass by about 1.8 meters. The initial
velocity is therefore v=\sqrt{2gh}\approx 6 m/s.

Escape velocity from earth is 11 km/s and the diameter of earth is 12,750 km.
The densities of asteroids is not at all well known (see
the Wikipedia article, “Standard asteroid physical characteristics”
but a reasonable estimate might be about 4/9 the density of earth.

So the diameter of the asteroid would be about
{}12750\cdot(6/11000)\cdot(3/2)\ \approx 10 km.

• 10.

What is the length of the y=x^{2} parabola in the region -1\leq x\leq 1?
Please include all calculations.

Answer:

\int_{-1}^{1}\sqrt{1+4x^{2}}\>dx=\sqrt{5}+\frac{1}{2}\sinh^{-1}(2)=2.95789

• 11.

Approximately how large a supply of antimatter would be needed, in
order to propel a spacecraft with the mass of the International Space
Station into orbit around Proxima Centauri, in one year as experienced
by its crew?
Please include all calculations.

Answer: 706,000 kg

Explanation:
We’ll assume that the Space Station is accelerated rapidly to its cruising
speed, and we’ll ignore the need to decelerate when it reaches Proxima
Centauri.

The first step is to compute the velocity. We’ll use the speed of light as the
unit of speed and a year as the unit of time. Proxima Centauri is 4.25 light years
away. If the spaceship travels at speed v, then as measured from earth,
the time required \tau=4.25/v. Because of time dilation, this appears to
the crew as \tau\sqrt{1-v^{2}}. So we have the equation
{}1=(4.25/v)\sqrt{1-v^{2}}, so
{}1=(4.25^{2}/v^{2})(1-v^{2}), so
v^{2}-18.0625(1-v^{2}), so v^{2}=18.0625/19.062=0.9475 and
v=0.9734.

If the Space Station has mass M, then its total energy at speed v is
M/\sqrt{1-v^{2}}, so the additional energy is
M/\sqrt{1-v^{2}}-M (c is still 1). If we annihilate mass m of
antimatter with mass m of matter, then that liberates 2m of energy
We have {}2m=M\cdot((1/\sqrt{1-v^{2}})-1)=3.3661M so m=1.683M. Taking m
to be 420,000 kg, that gives a value of 706,000 kg of antimatter.

• 12.

Approximately how many errors will a standard laptop suffer over
its lifetime, due to cosmic rays hitting the microchip?
Please include all calculations.

Answer: Estimates vary widely, but one commonly cited figure is
about 1 error per 256 megabytes of RAM per month. Assuming an 8 GByte laptop and
a 5 year lifespan, that will total 1920 errors.

• 13.

What is the approximate probability that a randomly-chosen
100-digit integer is prime?
Please include all calculations.

Answer: Approximately {}1/\log_{e}(5.5\cdot 10^{99})=1/(\log_{e}(5.5)+99\cdot\log_{e}(10))=0.00435. That should be accurate to within 1%.

• 14.

What is the probability that a randomly-chosen 100 × 100 matrix,
over the finite field F_{2}, is invertible?
Please include all calculations.

Answer: 0.289

Explanation: Calculate the probability that
the 2nd row is linearly independent from the first, then the
probability that the 3rd row is outside the span of the first two, the
probability that the 4th row is outside the span of the first three,
etc. and multiplying them all together. In backward order this is

{}1/2\cdot(3/4)\cdot(7/8)\cdot(15/16)\cdot...\approx 0.289.

• 15.

How does the total weight of all the uranium that humans mined,
compare to the total weight of all the gold that they’ve mined?
Please include all calculations.

Answer: It is estimated that humans have mined about 209,000 tones of
gold and about 2.8 million tons of uranium, a difference of a factor of
about 13.

Explanation: The figure for gold comes from
the World Gold Council. The figure for uranium was estimated by the authors based on the
information on
this web page of the World Nucler Association

• 16.

What is the Shannon entropy of a positive integer n that’s chosen
with probability Pr[n] = {}6/(\pi^{2}\cdot n^{2})?
Please include all calculations.

Answer: 2.362

Matlab:

ent=0;
for i=1:1000000
p = 6/((pi^2)*(i^2));
ent = ent - p*log(p);
sum = sum+p;
end
ent/log(2)

Revised version of problem:
On consideration, after running the above, we thought that perhaps the wording
of the problem was unfair to GPT. After all, the answer probably has no
elegant closed-form expression, and it is not easy to get a highly precise
numerical answer (the convergence rate of the naïve summation is
O(\log(n)/n)). So we tried the two systems with the following revised
question:

Compute the Shannon entropy of a positive integer n that’s chosen
with probability Pr[n] = {}6/(\pi^{2}\cdot n^{2}) to 3 digit accuracy.
Please include all calculations.

• 17.

Assume that IQs are normally distributed, with a mean of 100 and a
standard deviation of 15. For which n does the number of people with
IQ n exceed the number of people with IQ n+1 by the maximum amount?
Please include all calculations.

Answer: Approximately n=14.5

Explanation:
Let f(x) be the probability density function for the normal distribution
with mean 100.

f(x)=\frac{1}{\sqrt{2\pi}\cdot 15}\mbox{exp}(-(x-100)^{2}/2\cdot 15^{2})

At the infinitesimal scale, the change between people with IQ x and people
with IQ x+\epsilon is maximized when the derivative of the probability
distribution is negative and its magnitude is maximal; thus, at a value x
greater than the mean where the second derivative is 0.

The first derivative of f,
f^{\prime}(x)=-((x-100)/15^{2})f(x).

Setting the second derivative of f to be zero, we have,
f^{\prime\prime}=((x-100)^{2}/15^{2})^{2}-(1/15^{2})f(x), so
(x-100)^{2}=15^{2}, which has two roots: x=85, where the first
derivative reaches a maximum positive value and x=105 where it reaches
a maximum magnitude negative value.

Now, that does not directly give us the value of x that maximizes
f(x)-f(x+1), and finding the true maximum would involve solving
an inelegant transcendental equation.
However, since the third derivative of f is fairly small and
the second derivative is 0, it is safe to assume that picking x and x+1
evenly across x=15 gives values close to the maximum value. Using binary
search, one can determine that the true maximum is 114.5028.

• 18.

Suppose a randomized algorithm outputs the correct answer (yes or
no) with probability 0.9. If we take the majority answer out of 100
independent runs, with what probability will the answer be wrong?
Please include all calculations.

Answer: {}3.232\cdot 10^{-24}.

Explanation: Assume that if the results are exactly 50/50, then
you flip a fair coin. Then the exact value is

(1/2)\cdot C(100.50)\cdot 0.9^{50}\cdot 0.1^{50}+\sum_{i=1}^{50}C(100,50+i)\cdot 0.9^{50-i}\cdot 0.1^{50+i}

However, since this is (approximately)
a geometrically decreasing series with ratio of 1/9,
it suffices to take the first five terms to get three digit accuracy.

Matlab: (Matlab warns that its value may have a relative error of
{}10^{-14}).

p=nchoosek(100,50)0.9^500.1^50
sum=p/2
for i=1:5
p=nchoosek(100,50+i)*0.9^(50-i)*0.1^(50+i)
sum = sum+p
end
sum

• 19.

In a Manhattan-like two-dimensional grid, how many distinct
shortest paths are there that go from a fixed starting point to a
fixed endpoint 5 blocks to the north and 5 blocks to the east?
Please include all calculations.

Answer: 252.

Explanation: You have to take 10 steps: 5 of them north and 5 of
them east. They can be in any order. So the number is C(10,5) = 252.

• 20.

When 3-dimensional spheres are packed in their densest possible
configuration, what percentage of the space is empty?
Please include all calculations.

Answer: {}1-\pi/3\sqrt{2}=0.2595. See the
Wikipedia article
“Kepler conjecture”.

Where this page came from

This page was imported from arXiv. “Testing GPT-4 with Wolfram Alpha and Code Interpreter plug-ins on math and science problems” by Ernest Davis, Scott Aaronson, arXiv:2308.05713 (2025), published under CC BY 4.0. Changed here: set as a page from arXiv's HTML version; footnotes and author notes left out; figures the paper marks as under other terms left out.

Nobody has written it yet — it is the source material at a new address, which is why search engines are asked to skip it and why no one earns from it. It is up for grabs: take it on, and it is yours to rewrite and to earn from.