Locality Does Not Imply Reachability: Boundary Repair in Block-Sparse Causal Attention
About
Sparse causal attention is usually described by sequence locality: nearby tokens should remain easy to access, while distant tokens may be dropped to reduce cost. This paper studies a mismatch between sequence locality and attention-graph reachability. In fixed block causal attention, two adjacent tokens can be disconnected in the attention graph at every depth. We formalize this boundary artifact through structural dependency sets: if every attention layer uses the same fixed block causal mask and all remaining operations are positionwise, a target representation can depend only on tokens in its own block prefix. This yields an architecture-level boundary-copy separation for a constructed K-way boundary-copy distribution, with top-1 accuracy upper bound 1/K and expected cross-entropy lower bound log K. We then derive phase-conditioned coverage functions showing that reachability depends on both source-target distance and the target's offset within its block. These coverage laws predict when a sparse pattern should fail, when a repair can help, and why sliding-window attention and boundary repair are not interchangeable. Boundary Bridge Attention is treated as a constructive witness: it preserves the fixed block path and adds zero-additional-parameter auxiliary causal edges near block boundaries using shared projections. Controlled 1024-token experiments show that gains concentrate in coverage-aligned diagnostics. As secondary external-validity evidence, a fixed-checkpoint 8K-token Qwen2.5-7B probe shows the same coverage-incomparability pattern. The contribution is a theory-guided diagnostic framework for locality-reachability mismatch in block-sparse causal attention, together with phase-conditioned coverage analysis and a minimal constructive repair.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Candidate-ranking Retrieval | Needle 8192-token synthetic probes | Accuracy57.8 | 8 | |
| Candidate-ranking Retrieval | Diagnostic Probes 1024-token context | Needle Accuracy43.8 | 6 | |
| Language Modeling | 1024-token context (val) | Best Perplexity (PPL)30.84 | 6 | |
| Candidate-ranking Retrieval | 8192-token synthetic probes Bridge-window | Accuracy59.9 | 4 | |
| Candidate-ranking Retrieval | Synthetic probes 8192-token Semantic exact | Accuracy50 | 4 | |
| Candidate-ranking Retrieval | synthetic probes Semantic paraphrase 8192-token | Accuracy45.8 | 4 | |
| Candidate-ranking Retrieval | 8192-token synthetic probes Semantic paraphrase+distractor | Accuracy14.6 | 4 | |
| Candidate-ranking Retrieval | 8192-token synthetic probes Weighted average | Accuracy50.2 | 4 |