Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension

About

Visual Text Comprehension (VTC) renders text into images for a vision-language model (VLM) to read, sidestepping LLM context-window limits and powering applications from long-page OCR to multi-page memory QA. Yet existing VTC pipelines treat rendering and layout as a fixed, content-agnostic preprocessing step and offer little mechanistic understanding of how VLMs internally process visualized text. Through a focused empirical study on VTC QA tasks, we reveal that VLMs exhibit a localization-without-utilization regime: evidence-localizing attention emerges sharply in the middle-to-late layers and is largely decoupled from answer correctness, yet simply enlarging the localized spans on the rendered page recovers a large fraction of the failures. Building on these observations, we propose AGAR (Attention-Guided Adaptive Rendering), a training-free, model-agnostic method that leverages a VLM's own middle-to-late layer attention to identify the top-K important visual patches, maps them back to word spans, and re-renders the page with those spans enlarged before re-inferring the answer. Extensive experiments across nine VTC benchmarks (short-form, long-context, and multi-page memory QA) and four VLM backbones show that AGAR (i)consistently improves off-the-shelf VLMs as a plug-and-play enhancement, (ii)composes with VLM post-training to yield further gains, and (iii)remains robust under both visual- and text-side input degradation.

Shenglai Zeng, Qirui Wang, Kai Guo, Xinnan Dai, Xianxuan Long, Hui Liu• 2026

Related benchmarks

TaskDatasetResultRank
Question AnsweringNewsQA
F160.1
67
Question AnsweringNQ
F167.8
42
Question AnsweringNewsQA (test)
F147.2
40
Short Answer Question AnsweringHotpotQA
F159.6
39
Question AnsweringHotpotQA distractor setting (test)--
34
Multi-page Memory-based Question AnsweringLoCoMo multi-page memory
SingleHop Accuracy44.5
27
Multi-page Question AnsweringLongBench multi-page
HQA F148.9
27
Short-form Question AnsweringTQA (TriviaQA) (test)
F1 Score68.4
18
Short Answer Question AnsweringNatural Questions (NQ) (test)
F1 Score67.4
16
Multi-page Visual Question AnsweringLongBench multi-page
HQA Score63.6
9
Showing 10 of 15 rows

Other info

Follow for update