Applicability of Large Language Models for Evaluating Digital Exercises in Higher Education

April 22nd, 2024

As universities strive to enhance the effectiveness of their lecture exercises, there arises a need for diverse and realistic test user scenarios to evaluate the understandability and usefulness of educational materials. However, in creating such scenarios a number of challenges arise: Real world students can rarely be used for testing, they are likely inexperienced or experienced but have little incentive to refresh their knowledge to test tasks for lectures. In addition, leaking tasks to students can lead to skewed learning results during the actual course. On the other hand, simple unit tests can neglect the importance of a task being understandable to a human and do not offer much insight into task difficulty.

Thesis Type
  • Bachelor
Presentation room
Seminar room I5 6202
Stefan Decker
Laurenz Neumann
Maximilian Kißgen

Thesis Objective: This Bachelor’s thesis aims to investigate the feasibility and effectiveness of LLMs posing as artificial test users for university lecture exercises. The goal is to create personas capable of engaging with educational content in a manner reflective of real-world student behaviour.

Thesis Goals:

  1. Developing pipelines to generate LLM test users based on technical documentation and lecture content.
  2. Assessing the adaptability of LLM-generated test users across different subject domains.
  3. Evaluating the usefulness of LLM test users for providing feedback on educational content and preventing possible misunderstanding of tasks.
  4. Generating reproducible LLM test user personas for reliable testing.

  • Background in machine learning, natural language processing, and/or educational technology.
  • Proficiency in programming languages such as Python and experience with relevant libraries (e.g., TensorFlow, PyTorch).
  • Experience with Jupiter notebooks and CI/CD.