Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

GigaCheck: Detecting LLM-generated Content via Object-Centric Span Localization

About

With the increasing quality and spread of LLM assistants, the amount of generated content is growing rapidly. In many cases and tasks, such texts are already indistinguishable from those written by humans, and the quality of generation continues to increase. At the same time, detection methods are advancing more slowly than generation models, making it challenging to prevent misuse of generative AI technologies. We propose GigaCheck, a dual-strategy framework for AI-generated text detection. At the document level, we leverage the representation learning of fine-tuned LLMs to discern authorship with high data efficiency. At the span level, we introduce a novel structural adaptation that treats generated text segments as "objects." By integrating a DETR-like vision model with linguistic encoders, we achieve precise localization of AI intervals, effectively transferring the robustness of visual object detection to the textual domain. Experimental results across three classification and three localization benchmarks confirm the robustness of our approach. The shared fine-tuned backbone delivers strong accuracy in both scenarios, highlighting the generalization power of the learned embeddings. Moreover, we successfully demonstrate that visual detection architectures like DETR are not limited to pixel space, effectively generalizing to the localization of generated text spans. To ensure reproducibility and foster further research, we publicly release our source code.

Irina Tolstykh, Aleksandra Tsybina, Sergey Yakubson, Aleksandr Gordeev, Vladimir Dokholyan, Maksim Kuprashevich• 2024

Related benchmarks

TaskDatasetResultRank
Boundary DetectionRoFT-chatgpt GPT-3.5-turbo generated (test)
Accuracy67.65
34
ClassificationCoAuthor
Kappa0.4158
11
Boundary DetectionRoFT
Accuracy64.63
10
Machine-generated text detectionGhostbusters News
F1 Score100
10
Machine-generated text detectionGhostbusters Creative Writing
F1 Score100
10
Machine-generated text detectionGhostbusters Student Essays
F1 Score100
10
AI-generated text detectionMixSet 1.0 (test)
Average Score99
9
Machine-generated text detectionMAGE Unseen Domains & Unseen Model (test)
Human Recall95.65
9
Authorship Boundary DetectionRoFT chatgpt (leave-one-out cross-domain)
Accuracy (Speeches)0.5
7
Machine-generated text detectionMAGE Arbitrary-domains & Arbitrary-models (test)
Human Recall0.9572
5
Showing 10 of 19 rows

Other info

Follow for update