ChatEd: A Course Chatbot That Answers From the Instructor’s Own Materials

Figure 3 from Wang, Ramos and Lawrence: the ChatEd chatbot architecture (CC BY 4.0; reproduced unchanged).

Ask a general-purpose chatbot "When are the midterms?" and it cannot know. ChatEd, a chatbot built by researchers to help students in higher education, can, because it answers from the instructor's own course materials. The paper by Kevin Wang, Jason Ramos and Ramon Lawrence describes how it works and how it did against ChatGPT.

The problem

Large language models such as ChatGPT can give personalized help, but the authors note they can produce incorrect, misleading or biased answers, rarely give references to original sources, and lack knowledge of a particular course. Earlier course chatbots, such as Jill Watson, needed training on past questions and answers for each course, which was costly for individual instructors and limited their conversations.

How ChatEd works

An instructor uploads course materials (PowerPoint or PDF notes, web pages, posts and announcements) or gives their addresses. The system indexes them, for example by slide, in a PostgreSQL database with a similarity search library, Faiss, and each course or chatbot has its own database. No model training is needed.

When a student asks a question, ChatEd first searches the instructor's materials for similar passages. It then sends the question, the matching passages and the earlier chat to ChatGPT (the 3.5 turbo API, through the LangChain framework), which writes the answer, and ChatEd attaches the sources. The authors' three requirements are that answers be correct, relevant to the course and verifiable by students from the sources shown. It can plug into a learning management system or a discussion forum.

Figure 1 from the paper: the chatbot interface and an example conversation (CC BY 4.0; reproduced unchanged).

The test

The team tested it on questions from two database courses at a research-intensive university, an introduction to databases and a database implementation course, whose slides, labs and syllabi are on GitHub. The question set had 60 questions of three kinds: general ("Can each table have multiple foreign keys?"), domain-specific ("Are 'chunk' and 'block' interchangeable terms?") and managerial ("Is the exam open-book?"). The course instructor and teaching assistants graded answers on a scale of 1 to 5. For a sample of 20 questions, ChatEd and ChatGPT 3.5 scored:

Reviewers said ChatEd handled course-management questions that ChatGPT cannot answer, gave more concise and course-specific answers, and, for one question about errors when retrieving results, listed the 5 common errors covered in class where ChatGPT gave 12 bullet points.

A separate test of conversation, with follow-up questions and switches of topic, gave ChatEd average scores of 4.19 for relevancy, 5.00 for accuracy and 4.06 for helpfulness. Its weak spots: asked "Why do we need this?" after "What is a hash function?", it answered about a different topic; the authors guess the course documents confused it. It also could not reliably follow references to questions far back in the conversation.

What the authors conclude

The advantage is that instructors need only supply documents; the system uses the language model's general knowledge and conversation while keeping the local context and source references. The improvement over ChatGPT depends on the material provided: ChatGPT does well on general questions with accepted answers online and worse on course-specific or administrative ones such as deadlines and policies. The authors plan tests in more courses, better question answering and making the system available to instructors.

Sources

• Kevin Wang, Jason Ramos and Ramon Lawrence, ChatEd: A Chatbot Leveraging ChatGPT for an Enhanced Learning Experience in Higher Education, arXiv:2401.00052 (2023), licensed CC BY 4.0. Changed here: condensed and rewritten in hubnx's own words; evaluation criteria lists, the appendix tables and references are left out; the figures shown are the paper's own, unchanged. This version stays CC BY 4.0.