Study on relationship between adversarial texts and language errors: a human-computer interaction perspective

Large language models (LLMs) are widely applied in many human-computer interactive applications, such as chatbots. However, a deep understanding of the vulnerability of LLMs against adversarial attacks and language errors poses a threatening challenge. This study, therefore, presents a systematic analysis to disclose the relationships among language errors, adversarial texts, and LLMs. Each LLM is measured in language understanding ability and robustness within a human-computer interaction context. To further disclose the differences between language errors and adversarial texts, we measured each LLM under 6 metrics, including Levinstein edit distance, Modification rate, Cosine similarity, perplexity, error rate, and BLEU. Through detailed experiments, we first prove that both language error texts and adversarial texts have a serious impact on the performance of LLMs. The quantified measure of the difference between these two texts is innovative in differentiating language errors and adversarial texts from clean texts.