Artificial Intelligence as a Tool to Evaluate Corporate Sustainability Reporting

In contrast to Financial Reporting, sustainability reporting mainly consists of unstructured information such as narrative and semi-narrative comments as well as semi-structured forms like individual measures for different, often enterprise-specific key performance indicators (KPI) in different units. Researchers have proposed various techniques to make sustainability reports accessible for quantitative studies: Natural language processing or text mining (NLP) derives structured information from textual content and calculates indicators for e.g. readability, complexity, tonality, and trustworthiness of the underlying documents. From a technical perspective, the calculation of linguistic indicators or the application of the bag of words method requires both specialised software and skilled personnel with linguistic and coding proficiency. The availability of artificial intelligence technologies (AI) is expected to pave the way for new and improved tools for NLP that deliver quicker results with less software and personnel requirements. From a conceptual view, AI should be capable of analysing vast amounts of unstructured data and derive meaningful KPIs for comparison. Our paper aims to evaluate the current performance of common AI tools with regard to NLP. To do so, we define test samples from a sustainability report, calculate a readability index and perform different analyses with the bag of words method manually with traditional standard software. We then assign Chat GPT, Google Gemini and Microsoft Copilot with the same task for comparison. The results document the poor performance of the status-quo AI systems and show they are currently not able to provide a meaningful self-assessment with regard to the validity and limits of the results they provide.