Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Locality Does Not Imply Reachability: Boundary Repair in Block-Sparse Causal Attention

About

Sparse causal attention is usually described by sequence locality: nearby tokens should remain easy to access, while distant tokens may be dropped to reduce cost. This paper studies a mismatch between sequence locality and attention-graph reachability. In fixed block causal attention, two adjacent tokens can be disconnected in the attention graph at every depth. We formalize this boundary artifact through structural dependency sets: if every attention layer uses the same fixed block causal mask and all remaining operations are positionwise, a target representation can depend only on tokens in its own block prefix. This yields an architecture-level boundary-copy separation for a constructed K-way boundary-copy distribution, with top-1 accuracy upper bound 1/K and expected cross-entropy lower bound log K. We then derive phase-conditioned coverage functions showing that reachability depends on both source-target distance and the target's offset within its block. These coverage laws predict when a sparse pattern should fail, when a repair can help, and why sliding-window attention and boundary repair are not interchangeable. Boundary Bridge Attention is treated as a constructive witness: it preserves the fixed block path and adds zero-additional-parameter auxiliary causal edges near block boundaries using shared projections. Controlled 1024-token experiments show that gains concentrate in coverage-aligned diagnostics. As secondary external-validity evidence, a fixed-checkpoint 8K-token Qwen2.5-7B probe shows the same coverage-incomparability pattern. The contribution is a theory-guided diagnostic framework for locality-reachability mismatch in block-sparse causal attention, together with phase-conditioned coverage analysis and a minimal constructive repair.

Zhibo Yang• 2026

Related benchmarks

TaskDatasetResultRank
Candidate-ranking RetrievalNeedle 8192-token synthetic probes
Accuracy57.8
8
Candidate-ranking RetrievalDiagnostic Probes 1024-token context
Needle Accuracy43.8
6
Language Modeling1024-token context (val)
Best Perplexity (PPL)30.84
6
Candidate-ranking Retrieval8192-token synthetic probes Bridge-window
Accuracy59.9
4
Candidate-ranking RetrievalSynthetic probes 8192-token Semantic exact
Accuracy50
4
Candidate-ranking Retrievalsynthetic probes Semantic paraphrase 8192-token
Accuracy45.8
4
Candidate-ranking Retrieval8192-token synthetic probes Semantic paraphrase+distractor
Accuracy14.6
4
Candidate-ranking Retrieval8192-token synthetic probes Weighted average
Accuracy50.2
4
Showing 8 of 8 rows

Other info

Follow for update