Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Recovering Stranded Discrimination in Knowledge Tracing: Per-Item Bias Correction via Empirical-Bayes Shrinkage

About

Deployed knowledge-tracing models are typically frozen after training, yet systematic per-item logit bias arises, from limited per-item expressivity in backbone architectures and from post-deployment shifts in item properties, degrading prediction quality. Global post-hoc calibrators such as Platt scaling, temperature scaling, and isotonic regression improve probability estimates but leave discriminative ability, as measured by AUC, unchanged. This AUC invariance is a structural consequence of monotone score-only transforms; recovering the stranded discrimination requires conditioning on item identity. We propose SLC (State-space Logit Correction), which converts binary observations to Gaussian pseudo-observations via Laplace/IRLS, applies empirical-Bayes shrinkage through a Kalman smoother, and fits an offset-Platt link. The state-space formulation also yields a detectability bound that characterizes the Bernoulli information floor, explaining why temporal tracking provides no benefit at current data densities. Across four datasets, five backbones, and three seeds, SLC improves AUC on all four datasets and NLL on three, with the advantage concentrating on sparse items. Cross-domain controls suggest that the same phenomenon can arise beyond education when the deployed backbone leaves entity-level bias.

Xiaoran Yan, Cheng Tang, Atsushi Shimada• 2026

Related benchmarks

TaskDatasetResultRank
Knowledge TracingASSISTments 2017
AUC0.7182
31
Flight Delay PredictionFlight-delay (test)
AUC60.38
12
Binary ClassificationMovieLens 1M (temporal split)
AUC0.7951
10
Knowledge TracingALGEBRA
AUC83.25
9
Knowledge TracingEedi
AUC75.79
9
Knowledge TracingAS09
AUC0.7066
9
Knowledge TracingALGEBRA
NLL0.326
8
Knowledge TracingAS 17
NLL0.593
8
Knowledge TracingEedi
NLL0.552
8
Knowledge TracingAS09
NLL0.596
8
Showing 10 of 10 rows

Other info

Follow for update