Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension
About
Visual Text Comprehension (VTC) renders text into images for a vision-language model (VLM) to read, sidestepping LLM context-window limits and powering applications from long-page OCR to multi-page memory QA. Yet existing VTC pipelines treat rendering and layout as a fixed, content-agnostic preprocessing step and offer little mechanistic understanding of how VLMs internally process visualized text. Through a focused empirical study on VTC QA tasks, we reveal that VLMs exhibit a localization-without-utilization regime: evidence-localizing attention emerges sharply in the middle-to-late layers and is largely decoupled from answer correctness, yet simply enlarging the localized spans on the rendered page recovers a large fraction of the failures. Building on these observations, we propose AGAR (Attention-Guided Adaptive Rendering), a training-free, model-agnostic method that leverages a VLM's own middle-to-late layer attention to identify the top-K important visual patches, maps them back to word spans, and re-renders the page with those spans enlarged before re-inferring the answer. Extensive experiments across nine VTC benchmarks (short-form, long-context, and multi-page memory QA) and four VLM backbones show that AGAR (i)consistently improves off-the-shelf VLMs as a plug-and-play enhancement, (ii)composes with VLM post-training to yield further gains, and (iii)remains robust under both visual- and text-side input degradation.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Question Answering | NewsQA | F160.1 | 67 | |
| Question Answering | NQ | F167.8 | 42 | |
| Question Answering | NewsQA (test) | F147.2 | 40 | |
| Short Answer Question Answering | HotpotQA | F159.6 | 39 | |
| Question Answering | HotpotQA distractor setting (test) | -- | 34 | |
| Multi-page Memory-based Question Answering | LoCoMo multi-page memory | SingleHop Accuracy44.5 | 27 | |
| Multi-page Question Answering | LongBench multi-page | HQA F148.9 | 27 | |
| Short-form Question Answering | TQA (TriviaQA) (test) | F1 Score68.4 | 18 | |
| Short Answer Question Answering | Natural Questions (NQ) (test) | F1 Score67.4 | 16 | |
| Multi-page Visual Question Answering | LongBench multi-page | HQA Score63.6 | 9 |