Can large language models (LLMs) replace human reading? Empirical evidence from sustainability reports

This study explores the divergence between human reading and Large Language Model (LLM)-based reading in the context of sustainability reports. To capture this distinction, a saliency map representing human gaze is computed, illustrating how attention is directed not only by textual content but also by visual elements such as images, charts, and layout, which are typically overlooked by LLMs. The research first adopts an LLM-based framework to assess firms’ stated commitment to Environmental, Social, and Governance (ESG) practices as conveyed in their sustainability reports. The resulting LLM-generated scores are then empirically tested for their association with firm’s ESG disclosure rating evaluated by a third-party. In the second stage, the study examines how visual salience moderates the relationship between LLM-based reading and ESG disclosure rating. The findings reveal that: (1) LLMs can effectively extract ESG performance-related signals from reports; and (2) discrepancies between human and LLM-based reading, captured through visual salience, significantly influence the effectiveness of LLM-based assessments. These results suggest that while LLMs can be a powerful tool for analyzing ESG disclosures, their limitations in handling visual information warrant careful consideration in contexts where visual elements play a key communicative role.