Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

About

Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generation systems increasingly rely on structured inputs: source code, developer-tool output, markdown documents, tables, and repository metadata. We introduce a unified benchmark for span-level hallucination detection over code, tool output, structured documents, and existing natural-language RAG datasets. The benchmark is built by starting from grounded correct answers, injecting localized hallucinations with exact character labels, and validating the code test split with evidence-based review. Our fine-tuned Qwen3.5-2B detector reaches 0.689 span-F1 on the unified test set and 0.60 on the code-agent source, where it substantially outperforms LettuceDetect-large (0.17) and the strongest zero-shot LLM judges we evaluated (at most 0.22). The same model remains competitive on established natural-language benchmarks, with 81.8 RAGTruth example-F1 and 0.724 English PsiloQA IoU.

\'Ad\'am Kov\'acs, Bowei He, Xue Liu, Istv\'an Boros, Szilveszter T\'oth, G\'abor Recski• 2026

Related benchmarks

TaskDatasetResultRank
Span-level Hallucination DetectionLettuceDetect (test)
F1 Score86.6
12
Hallucination DetectionCode-agent (test)
Precision61.9
11
Hallucination DetectionRAGTruth Ex.-F1
Example-F181.8
5
Span-level Hallucination DetectionLettuceDetect Code-agent (test)
F1 Score60.2
4
Span-level Hallucination DetectionLettuceDetect Wikipedia (test)
F1 Score81.7
4
Hallucination DetectionPsiloQA EN (IoU)
IoU72.4
3
Hallucination DetectionPsiloQA 14-lang IoU
IoU68.9
3
Span-level Hallucination DetectionLettuceDetect ALL (test)
F1 Score68.9
3
Span-level Hallucination DetectionPsiloQA (test)
F1 Score73.2
3
Span-level Hallucination DetectionRAGTruth (test)
F1 Score57.4
3
Showing 10 of 10 rows

Other info

Follow for update