Collaborative inspection integrating both advanced LLM and traditional color difference quantification: A case study in infrared-based defect detection

An average annual power loss of 18.9% caused by module faults underscores the need for defect detection, especially as global photovoltaic (PV) capacity surpasses the terawatt scale. Existing data-driven approaches face a persistent precision–recall trade-off: supervised models require extensive labeled data, unsupervised methods are often unstable in complex backgrounds, and multimodal large language models (LLMs) can suffer from domain mismatch and largely qualitative reasoning. Prompt engineering can mitigate domain gaps but cannot quantify visual features; thermal defects appear as pixel-level deviations that can be quantified via Euclidean-distance analysis. However, neither approach alone resolves the precision–recall trade-off, motivating an integrated collaborative mechanism. Therefore, we propose a prompt-driven Hybrid LLM-Integrated Detection Framework (HLIDF) consisting of three modules: (1) a Prompt Optimization Module to mitigate domain misalignment; (2) a Dual-LLMs Parallel Analysis Module that runs precision-focused and recall-focused LLMs in parallel with cross-verification; and (3) a Quantitative Color Difference Decision Module that resolves conflicts through pixel-wise Euclidean-distance analysis. On the PVF-10 dataset for binary fault detection, HLIDF achieves 95.67% accuracy with 96.04% precision, 97.76% recall, and a 96.89% F1-score without fine-tuning, reducing the recognition error rate by 53.09% relative to a supervised ViT baseline and by 50.46% relative to a naïve LLM baseline. As one of the first few studies applying multimodal LLMs to infrared-based PV defect detection, this work establishes a practical framework for industrial deployment under weak supervision. For a 10 MW PV system generating 13,000 MWh annually, under typical assumptions on detectable fault-induced losses and electricity value, HLIDF can preserve over 105,000 USD in annual energy value.