Can GPT-4 Write Python Quiz Questions as Well as Instructors? A 1,100-Question Study

Writing good multiple-choice questions (MCQs) takes expertise and time, and the demand for fresh ones is high. A paper by Jacob Doughty, Zipiao Wan, Anishka Bompelli and colleagues asks whether GPT-4 can help with Python courses in higher education. The authors compared 651 questions generated by an LLM-powered system with 449 written by people, all aligned to 246 learning objectives (LOs) from six courses. This is a summary of the paper and its claims.

MCQ generation pipeline. The user input is processed, the Bloom’s taxonomy level of the learning objective is predicted, and that determines the question type and example questions. These are combined with the user input and design resources into the prompts sent to GPT-4, which outputs the stem, key and distractors. From the paper (Figure 1), CC BY 4.0.

How the questions were made

The system asks the user only for high-level information: a course title, a short course description with course-level objectives, the list of modules, and the module-level LO to be tested. Many earlier systems start from a passage of text instead. Here the questions are tied to objectives, so they can be checked against what students are meant to achieve.

Behind the prompt are static design resources: principles for good MCQs (for example, that distractors should be plausible and few, often just two), definitions of the six levels of the revised Bloom’s taxonomy, five types of programming question (recall, fill in the blank, correct output, code analysis and scenario-based), example questions for each type, and a JSON output format. A BERT classifier, fine-tuned on 21,380 objectives from 5,558 university courses, predicts each objective’s Bloom level. That level selects which question types to generate. The model was GPT-4 (gpt-4-0613) at its default temperature of 1.0, limited to 2,000 tokens, and it produced the stem, correct answer and distractors in a single step.

The data and the rubric

The 246 objectives came from four Python programming courses and two introductory data science courses. Three of the courses supplied human-written questions, assumed to be created by hand: 529 were collected, 51 left out because they could not be matched to a single objective and 29 because they had more than one correct choice, which left 449. For each objective the team generated one question per question type mapped to its Bloom level, or all types when no level could be assigned, for 651 in total.

Seven students and six computer science instructors, all authors, rated all 1,100 questions on six rubric items: sufficient information, a correct answer, unique choices, no obviously wrong choice, correct code, and alignment with the objective. The raters tried to answer each question first. Each question got at least one student and one instructor rating, 3,076 ratings in all. Disagreements were settled by majority vote, then instructor precedence, then precedence of whoever answered correctly, then the least favorable rating. Agreement was fair by Fleiss’ kappa (0.22), a statistic the authors say understates agreement when one label dominates; by Gwet’s AC1, which handles that, it ranged from 0.62 to 0.96.

MCQ evaluation results. The rubric is on the left; on the right, the generated MCQs (GPT) are compared with the human-crafted ones, with the five quality items (RQ1) above and the objective-alignment item (RQ2) below. From the paper (Figure 4), CC BY 4.0.

Results

Quality. Generated questions looked comparable to human ones on sufficient information, clear language and correct code. They were weaker on having one correct answer and on distractors. 4.9% of generated questions had more than one correct choice, against 1.1% of the human ones, and 4.0% had distractors that gave the answer away, against 0.9%. Both differences were statistically significant (p = 0.002 each). The authors found multiple correct answers mostly at the Apply and Create levels; in one example, three ways of converting a string to a list and reversing it all worked, but only one was marked correct. Answers given away were mostly in the stem, especially for fill-in-the-blank and scenario questions. In all, 81.7% of generated questions passed every criterion, which the authors read as fewer than one in five needing an instructor’s edit.

Alignment. Generated questions were noticeably better aligned with the objectives (p < 10⁻⁹). Human questions more often did not match the objective's cognitive level or were too different to serve as an assessment (20.5%, against 12.0% for generated), and more often did not relate to the objective at all (12.0%, against 4.8%). In one example the objective was to discuss why and how to write comments; the generated question asked why comments matter in Python code, while the human question was a true-or-false statement that clean code is easy to maintain. The authors suggest educators often align questions with a module’s topic rather than its objective, and note that they matched the human questions to objectives by hand, which may explain part of the gap.

Difficulty. Raters answered 71.5% (students) and 80.1% (instructors) of the generated questions correctly, against 62.6% and 76.3% for the human-written ones, so the human questions seemed harder. They were not told to try hard, so the authors urge caution.

What the authors conclude

LLM-powered tools can produce questions of comparable quality to those written by people, with better alignment, and could cut instructors’ workload if generated questions with problems are edited before use. The authors list limits: a small pool of raters, human questions written without knowing the rubric, no test in real classrooms, no comparison of difficulty, and a focus on Python in higher education. They plan to test the questions in real classes, study the novelty and diversity of generated questions, try GPT-4’s ability to refine questions with feedback, vary settings such as temperature, and extend the work to other programming languages.

Sources

• Jacob Doughty, Zipiao Wan, Anishka Bompelli et al., “A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education,” arXiv:2312.03173, licensed under CC BY 4.0. Changed here: summarized and rewritten in hubnx's own words; tables, prompt text, examples and references left out; figure captions shortened.