Understanding LLM Reasoning for Abstractive Summarization
About
Reasoning has substantially improved Large Language Models (LLMs) on analytical tasks such as mathematics and code generation, but its value for abstractive summarization remains unclear. To address this gap, we adapt general reasoning strategies to the summarization setting and conduct a large-scale comparative study of 8 reasoning strategies and 3 Large Reasoning Models (LRMs) across 8 diverse datasets, evaluating both summary quality and factual faithfulness. Our results show that reasoning is not a universal solution and its effectiveness depends strongly on the strategy and the summarization setting. In particular, we find a trade-off between summary quality and factual faithfulness. Explicit reasoning strategies often improve reference-based quality, but may weaken factual grounding, whereas implicit reasoning in LRMs shows the opposite tendency. We further find that increasing an LRM's internal reasoning budget does not reliably improve summarization and can even reduce factual consistency. These findings suggest that, for summarization, more reasoning is not always better. Effective reasoning should preserve faithful compression rather than induce over-elaboration. Our source code is publicly available.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Multi-document summarization | Multi-News (test) | -- | 45 | |
| Summarization | SamSum | BERTScore F190.57 | 30 | |
| Summarization | MultiNews (test) | Comprehensiveness4.98 | 24 | |
| Summarization | BookSum (test) | Comp Score5 | 24 | |
| Summarization | SciGen (test) | Completeness Score4.99 | 24 | |
| Summarization | Aggregate (test) | Comprehensiveness4.97 | 24 | |
| Summarization | arXiv (test) | Completeness Score5 | 24 | |
| Summarization | arXiv (test) | BS Score85.03 | 21 | |
| Abstractive Summarization | Multi-News 56k samples (test) | ROUGE Score20.72 | 12 | |
| Abstractive Summarization | CNN/DM sampled (test) | ROUGE Score22.86 | 12 |