Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Hindsight Credit Assignment for Long-Horizon LLM Agents

About

Large Language Model (LLM) agents often face significant credit assignment challenges in long-horizon, multi-step tasks due to sparse rewards. Existing value-free methods, such as Group Relative Policy Optimization (GRPO), encounter two fundamental bottlenecks: inaccurate step-level Q-value estimation and misaligned value baselines for intermediate states. To address these limitations, we introduce HCAPO, the first framework to integrate hindsight credit assignment into LLM agents. HCAPO leverages the LLM itself as a post-hoc critic to refine step-level Q-values through hindsight reasoning. Furthermore, HCAPO's multi-scale advantage mechanism effectively supplements the inaccurate value baselines at critical decision states. Evaluations across three challenging benchmarks, including WebShop and ALFWorld, demonstrate that HCAPO consistently outperforms state-of-the-art RL methods. Notably, HCAPO achieves a 7.7% improvement in success rate on WebShop and a 13.8% on ALFWorld over GRPO using the Qwen2.5-7B-Instruct model. These results indicate that HCAPO significantly enhances exploration efficiency, promotes concise decision-making, and ensures scalability in complex, long-horizon tasks.

Hui-Ze Tan, Xiao-Wen Yang, Hao Chen, Jie-Jing Shao, Yi Wen, Yuteng Shen, Weihong Luo, Xiku Du, Lan-Zhe Guo, Yu-Feng Li• 2026

Related benchmarks

TaskDatasetResultRank
Multi-hop Question Answering2Wiki--
152
Single-hop Question AnsweringPopQA--
104
Web Navigation and ShoppingWebshop
Success Rate73.8
81
Single-hop Question AnsweringTriviaQA--
81
Multi-hop Question AnsweringBamboogle
Accuracy69
62
Multi-hop Question AnsweringHotpotQA
Accuracy42.1
30
Household Agent InteractionAlfWorld
Pick Success Rate99.1
20
Question AnsweringSearch-augmented QA tasks Average
Average Accuracy48.3
12
Multi-hop Question AnsweringMuSiQue
Accuracy17.7
12
Showing 9 of 9 rows

Other info

Follow for update