KazakhTextDuplicates: A Controlled Multi-Regime Benchmark for Semantic Deduplication, Semantic Similarity, and Retrieval in Kazakh

Reliable evaluation resources for semantic deduplication, semantic textual similarity (STS), and retrieval remain limited for low-resource and morphologically rich languages. This study introduces KazakhTextDuplicates, a Kazakh-language benchmark for controlled evaluation of semantic duplication under varying levels of semantic preservation and surface-form distortion. The benchmark includes two complementary versions. KazakhTextDuplicates v1.0 is a diagnostic dataset derived from naturally occurring duplicate and near-duplicate pairs for analyzing relationships between duplication labels and lexical overlap. KazakhTextDuplicates v2.0 is a controlled benchmark created using deterministic transformations that define seven semantic duplication regimes: exact duplication, contextual reformulation, paraphrasing, partial semantic overlap, and three levels of character-level noise. Each pair is assigned both a regime label and a predefined similarity score, enabling evaluation across duplicate classification, STS, and retrieval tasks. Five embedding models (BGE-M3, LaBSE, Multilingual-E5-Large, KazEmbed-v5, and OpenAI text-embedding-3-large) were evaluated in a zero-shot setting. Results show that v1.0 is strongly affected by surface-form similarity and is therefore more suitable for diagnostic analysis. In contrast, v2.0 provides a more challenging and informative evaluation environment. OpenAI text-embedding-3-large achieved the strongest STS performance (Pearson = 0.510, Spearman = 0.652, MSE = 0.075) and the best duplicate regime classification results (Accuracy = 0.155, Macro-F1 = 0.066). Retrieval performance remained strong at higher cutoffs despite lower first-rank stability. The results demonstrate that benchmark design substantially affects semantic similarity evaluation and emphasize the need for controlled assessment in low-resource agglutinative languages.