CUDRT: Benchmarking the Detection Models of Human vs. Large Language Models Generated Texts

While large language models (LLMs) have greatly enhanced text generation across industries, their human-like outputs make distinguishing between human and AI authorship challenging. Although many LLM-generated text detectors exist, current benchmarks mainly rely on static datasets, limiting their effectiveness in assessing model-based detectors requiring prior training. Furthermore, these benchmarks focus on specific scenarios like question answering and text refinement and are primarily limited to English, overlooking broader linguistic applications and LLM subtleties. To address these gaps, we construct a comprehensive bilingual benchmark in Chinese and English to rigorously evaluate mainstream LLM-generated text detection methods. We categorize LLM text generation into five key operations-Create, Update, Delete, Rewrite, and Translate (CUDRT)-covering the full range of LLM activities. For each CUDRT category, we developed extensive datasets enabling thorough assessment of detection performance, incorporating the latest mainstream LLMs for each language. We also establish a robust evaluation framework to support scalable, reproducible experiments, facilitating an in-depth analysis of how LLM operations, different LLMs, datasets, and multilingual training sets impact detector performance, particularly for model-based methods. Our extensive experiments provide critical insights for optimizing LLM-generated text detectors and suggest future directions to improve detection accuracy and generalization across diverse scenarios.Source code and dataset are available at GitHub.