Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMs

About

LVLMs have achieved strong multimodal reasoning capabilities but remain prone to hallucinations, producing outputs inconsistent with visual inputs or user instructions. Existing training-free methods, including contrastive decoding and auxiliary expert models, which incur several times more computational overhead and may introduce potential interference, as well as static internal signal enhancement, are often vulnerable to the attention sink phenomenon. We find that internal Positive Attention Dynamics (PAD) in LVLMs naturally reveal semantically core visual regions under the distortions of attention sinks. Based on this, we propose Positive Attention Dynamics Enhancement (PADE), a training-free attention intervention that constructs a PAD map to identify semantically core visual regions, applies per-head Median Absolute Deviation Scaling to adaptively control the intervention strength, and leverages System-Token Compensation to maintain attention to complex user instructions and support long-term output consistency. Experiments on multiple LVLMs and benchmarks show that PADE improves visual grounding and reduces hallucinations, validating the effectiveness of leveraging internal attention dynamics for reliable multimodal reasoning.

Guangtao Lyu, Qi Liu, Chenghao Xu, Jiexi Yan, Muli Yang, Xueting Li, Fen Fang, Cheng Deng• 2026

Related benchmarks

TaskDatasetResultRank
Visual Question AnsweringVizWiz
Accuracy52.08
1525
Multimodal EvaluationMME--
658
Object HallucinationPOPE Adversarial
Accuracy85.12
288
Object HallucinationPOPE (Random)
F1 Score87.42
285
Object HallucinationPOPE Popular
F1 Score86.28
273
Multimodal EvaluationMM-Vet--
180
Hallucination EvaluationHallusionBench--
108
Object Hallucination in Open-ended CaptioningChair (test)
CHAIR_S51.8
50
Generative HallucinationAMBER Generative
Coverage (%)51.8
36
Multimodal EvaluationLLaVA-Bench-Wild (LLaVA-W)
Overall Score65.92
24
Showing 10 of 10 rows

Other info

Follow for update