Eye-Tracking Technology for Cognition-Adaptive LLM-Based Assistants at the Workplace

Large language models (LLMs) are playing an increasingly important role in knowledge work. LLM-based assistants will enable automation and augmentation of a broad spectrum of knowledge worker tasks in the future. Thereby, it should be noted that LLM-based assistants also come with impacts on their users. Previous automation research has shown that cognitive states play an important role in automated systems. For example, users are known to have difficulties in maintaining situation awareness and managing cognitive load. It can be assumed that cognitive states also play a key role in the interaction between humans and LLM-based assistants enabling automation and augmentation of tasks. Eye tracking is a promising technology to examine cognitive states of users when engaging with LLM-based assistants. In this book chapter, we present results from a pilot study investigating the cognition of users engaging with LLM-based reading assistants. On this basis, we introduce a new class of systems titled cognition-adaptive LLM-based assistants that emphasizes a cognitive perspective on LLM-based assistance from a user’s point of view. Specifically, we propose to leverage the capabilities of eye tracking to automatically recognize cognitive states of LLM-based assistance users and on this basis provide advanced cognition-adaptive support. We introduce design elements and a roadmap of future work towards cognition-adaptive LLM-based assistants.