Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Streaming Knowledge Compilation: Proactive Materiality-Scored Pinning for Time-Evolving LLM Wikis

About

LLM wiki systems compile knowledge into pre-filled KV caches for efficient inference, but assume a static corpus -- an assumption that fails whenever the underlying information landscape evolves. We formalize Streaming Knowledge Compilation: given a document stream, a fixed token budget, and future queries unknown at ingestion time, maintain a compiled wiki that minimizes cumulative regret against an offline oracle with perfect foresight. The enabling insight is a materiality signal $\phi_t(k,n)\in[0,1]$ that scores document importance for entity $k$ at time $t$, acting as a query-relevance surrogate for proactive pinning before queries arrive; we prove an $O(\sqrt{T\log K})$ regret bound where $\varepsilon=\mathbb{E}[|\phi_t-\hat\phi_t|]$ is the only domain-specific quantity. We instantiate in two domains: finance, where $\phi_t$ is abnormal stock volatility predicted by frozen Llama 3.1 8B classification head (AUROC = 0.728 on 76K articles, strict temporal split; $1.49\times$ higher realized forward volatility for predicted-material articles); and Wikipedia, where $\phi_t$ is the Abnormal Edit Ratio (AER), a cross-sectionally normalized edit velocity -- showing the same algorithm generalizes beyond the finance domain. End-to-end QA evaluation on 173 matched pairs (finance) and 119 (Wikipedia) reveals a pervasive LLM-as-judge confound on post-training knowledge, establishing that regret analysis -- not absolute QA scores -- is the reliable evaluation metric for compiled knowledge systems. Finance cumulative regret converges to -20.0 (-0.12/step); Wikipedia to +16.0 (+0.13/step), with the positive sign confirming that Wikipedia edit content is genuinely post-training -- richer context consistently improves scores (No Wiki 3.80 vs. Oracle 4.74) -- and eliminates this confound. The $O(\sqrt{T\log K})$ guarantee applies to any domain where knowledge gaps can be predicted from streaming signals.

Juan M. Huerta• 2026

Related benchmarks

TaskDatasetResultRank
End-to-end Question AnsweringEvent-driven queries 2025-2026
Step Regret0.00e+0
5
Question AnsweringWikipedia events 2024-2025 (test)
Standing Q_t^bg3.67
5
Question AnsweringWiki-based QA Finance Standing queries (Q^bg_t) base-wiki quality
Score (1-5)3.83
5
Question AnsweringWiki-based QA Finance Event-driven queries (Q^vol_t) pin-layer quality
LLM Judge Score (1-5)3.71
5
Question AnsweringWiki-based QA Finance (overall)
LLM-as-judge Score (1-5)3.79
5
Materiality PredictionFinance News (test)
AUROC72.9
4
Streaming Knowledge CompilationFinance instantiation
Matched Pairs173
1
Streaming Knowledge CompilationWikipedia instantiation
Matched Pairs119
1
Showing 8 of 8 rows

Other info

Follow for update