Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents
About
Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generation systems increasingly rely on structured inputs: source code, developer-tool output, markdown documents, tables, and repository metadata. We introduce a unified benchmark for span-level hallucination detection over code, tool output, structured documents, and existing natural-language RAG datasets. The benchmark is built by starting from grounded correct answers, injecting localized hallucinations with exact character labels, and validating the code test split with evidence-based review. Our fine-tuned Qwen3.5-2B detector reaches 0.689 span-F1 on the unified test set and 0.60 on the code-agent source, where it substantially outperforms LettuceDetect-large (0.17) and the strongest zero-shot LLM judges we evaluated (at most 0.22). The same model remains competitive on established natural-language benchmarks, with 81.8 RAGTruth example-F1 and 0.724 English PsiloQA IoU.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Span-level Hallucination Detection | LettuceDetect (test) | F1 Score86.6 | 12 | |
| Hallucination Detection | Code-agent (test) | Precision61.9 | 11 | |
| Hallucination Detection | RAGTruth Ex.-F1 | Example-F181.8 | 5 | |
| Span-level Hallucination Detection | LettuceDetect Code-agent (test) | F1 Score60.2 | 4 | |
| Span-level Hallucination Detection | LettuceDetect Wikipedia (test) | F1 Score81.7 | 4 | |
| Hallucination Detection | PsiloQA EN (IoU) | IoU72.4 | 3 | |
| Hallucination Detection | PsiloQA 14-lang IoU | IoU68.9 | 3 | |
| Span-level Hallucination Detection | LettuceDetect ALL (test) | F1 Score68.9 | 3 | |
| Span-level Hallucination Detection | PsiloQA (test) | F1 Score73.2 | 3 | |
| Span-level Hallucination Detection | RAGTruth (test) | F1 Score57.4 | 3 |