Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Rethinking Local Learning: A Cheaper and Faster Recipe for LLM Post-Training

About

LLM post-training typically propagates task gradients through the full depth of the model. Although this end-to-end structure is simple and general, it couples task adaptation to full-depth activation storage, long-range backward dependencies and direct task-gradient access to pretrained representations. We argue that this full-depth backward coupling can be unnecessarily expensive and intrusive, particularly when post-training supervision is much narrower than pre-training. To this end, we propose \textbf{LoPT}: Local-Learning Post-Training, a simple post-training strategy that makes gradient reach an explicit design choice. LoPT places a single gradient boundary at the transformer midpoint: the second-half block learns from the task objective, while the first-half block is updated by a lightweight feature-reconstruction objective to preserve useful representations and maintain interface compatibility. LoPT shortens the task-induced backward path while limiting direct interference from narrow task gradients on early-layer representations. Extensive experiments demonstrate that LoPT achieves competitive performance with lower memory cost, higher training efficiency and better retention of pretrained capabilities. Our code is available at: https://github.com/HumyuShi/LoPT

Hengyu Shi, Tianyang Han, Peizhe Wang, Zhiling Wang, Xu Yang, Junhao Su• 2026

Related benchmarks

TaskDatasetResultRank
Instruction FollowingIFEval--
836
Commonsense ReasoningHellaSwag
HellaSwag Accuracy86.24
711
Mathematical ReasoningGSM8K--
204
Massive Multitask Language UnderstandingMMLU
Accuracy83.34
129
Large Language Model EvaluationHuggingFace Open LLM Leaderboard lm-eval-harness default (various)
HellaSwag80.97
36
Commonsense ReasoningWinoGrande
Accuracy80.71
24
Truthfulness EvaluationTruthfulQA
Accuracy65.75
20
Language Model Evaluationlm-eval-harness (test)
MMLU74.18
9
Reasoning Question AnsweringARC Challenge
Accuracy74.55
3
Showing 9 of 9 rows

Other info

Follow for update