Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

Understanding LLM Reasoning for Abstractive Summarization

About

Reasoning has substantially improved Large Language Models (LLMs) on analytical tasks such as mathematics and code generation, but its value for abstractive summarization remains unclear. To address this gap, we adapt general reasoning strategies to the summarization setting and conduct a large-scale comparative study of 8 reasoning strategies and 3 Large Reasoning Models (LRMs) across 8 diverse datasets, evaluating both summary quality and factual faithfulness. Our results show that reasoning is not a universal solution and its effectiveness depends strongly on the strategy and the summarization setting. In particular, we find a trade-off between summary quality and factual faithfulness. Explicit reasoning strategies often improve reference-based quality, but may weaken factual grounding, whereas implicit reasoning in LRMs shows the opposite tendency. We further find that increasing an LRM's internal reasoning budget does not reliably improve summarization and can even reduce factual consistency. These findings suggest that, for summarization, more reasoning is not always better. Effective reasoning should preserve faithful compression rather than induce over-elaboration. Our source code is publicly available.

Haohan Yuan, Haopeng Zhang• 2025

Related benchmarks

TaskDatasetResultRank
Multi-document summarizationMulti-News (test)--
45
SummarizationSamSum
BERTScore F190.57
30
SummarizationMultiNews (test)
Comprehensiveness4.98
24
SummarizationBookSum (test)
Comp Score5
24
SummarizationSciGen (test)
Completeness Score4.99
24
SummarizationAggregate (test)
Comprehensiveness4.97
24
SummarizationarXiv (test)
Completeness Score5
24
SummarizationarXiv (test)
BS Score85.03
21
Abstractive SummarizationMulti-News 56k samples (test)
ROUGE Score20.72
12
Abstractive SummarizationCNN/DM sampled (test)
ROUGE Score22.86
12
Showing 10 of 42 rows

Other info

Follow for update