Currently, the primary way users interact with Large Language Models (LLMs) is through two-dimensional chat interfaces. However, for use cases in Extended Reality (XR) environments, the interaction paradigm shifts from a flat screen to a spatial experience. Here, LLMs can, e.g., be represented as XR agents, a personified version of the LLM. While 3D environments offer high potential for more immersive and intuitive interactions, they also introduce significant challenges regarding user interface design. Simply porting a 2D chat window into a 3D space often feels clunky or breaks the sense of presence, yet purely voice-based interaction may lack the precision or privacy that text provides. This is especially true for tutoring scenarios where the agents need to give precise and memorable instructions. There is currently a lack of systematic research on which interface modalities best support the strengths of LLMs while maintaining the immersion of an XR environment.
Thesis Type |
|
Student |
Tim Engels |
Status |
Running |
Proposal on |
24/06/2026 10:00 am |
Proposal room |
Seminar room I5 6202 |
Presentation room |
Seminar room I5 6202 |
Supervisor(s) |
Stefan Decker Wolfgang Prinz |
Advisor(s) |
Benedikt Hensen |
Contact |
hensen@dbis.rwth-aachen.de |
The goal of this thesis is to investigate and compare different interaction modalities for LLM-powered agents in XR to determine which method feels most natural and least intrusive for the user. To achieve this, the student will implement a simple LLM-driven agent within a virtual environment. The core of the work will be the development of a number of distinct interaction interfaces and possibly a combination of them. Potential modalities to be explored include, but are not limited to:
- Voice-to-Voice: Using Speech-to-Text (STT) and Text-to-Speech (TTS) for a hands-free, conversational experience.
- Spatial Text Bubbles: Visualizing dialogue through floating text elements anchored to the agent.
- Diegetic Interfaces: Utilizing an in-world virtual tablet or wrist-display for text-based input and output.
Once implemented, a user study will be conducted to evaluate these modalities. Participants will interact with the agent using the different interfaces to complete a set of predefined tasks. The study will focus on subjective metrics such as perceived naturalness, cognitive load, and level of immersion in the XR experience. Moreover, learning success in a learning scenario can be measured. By comparing these results, the thesis will aim to provide recommendations for the design of future AI-driven agents in spatial computing.
- Necessary: Proficiency in C#.
- Beneficial: Experience with Unity, knowledge of Mixed Reality (MR/VR/AR) development, and familiarity with conducting user studies or UX research.