LLM, Trust and Performance: case study of a Competitive Intelligence solution[LLM, confiance et performance : étude de cas d’une solution d’intelligence économique]

The recent democratization of Large Language Models (LLMs) has profoundly transformed access to and exploitation of external data, particularly in the field of Strategic and Competitive Intelligence (CI). While LLM-based solutions offer new opportunities to enhance decision-making, innovation detection, and information processing, they also raise significant challenges regarding their evaluation, notably in terms of performance, trust, and operational impact. Existing evaluation frameworks, largely inherited from traditional machine learning, appear insufficient to capture the multifaceted, subjective, and evolving nature of LLM-driven uses. This paper explores the evaluation of LLM-based CI solutions through an in-depth case study of a French data science project aimed at supporting innovation scouting for enterprises. Building on a review of state-of-the-art LLM evaluation metrics and methodologies, the study adopts an anthropocentric qualitative approach, based on eleven semi-structured interviews conducted with data and business stakeholders involved in the co-design of the solution. The analysis follows the CRISP-DM framework to examine how evaluation activities are embedded throughout the project lifecycle. The findings highlight an extreme complexity in defining and operationalizing evaluation criteria, driven by the coexistence of multiple objectives such as relevance, exhaustiveness, creativity, usability, and strategic impact. Beyond statistical metrics, the study reveals the growing importance of behavioral and psychological dimensions, particularly user trust, which strongly influence adoption and perceived value but remain costly and difficult to measure over time. Moreover, the rapid evolution of LLM technologies and the constraints related to sovereignty, confidentiality, and intellectual property further complicate model selection, benchmarking, and long-term governance. The paper concludes by discussing emerging challenges for performance assessment, project arbitration, and the balance between standardization and flexibility of practices. It calls for renewed evaluation frameworks that integrate technical, organizational, and human factors, and outlines avenues for future research on sustainable and interpretable evaluation of LLM-based CI systems.