Q-STRUM Debate: Query-Driven Contrastive Summarization for Recommendation Comparison
About
Query-driven recommendation with unknown items poses a challenge for users to understand why certain items are appropriate for their needs. Query-driven Contrastive Summarization (QCS) is a methodology designed to address this issue by leveraging language-based item descriptions to clarify contrasts between them. However, existing state-of-the-art contrastive summarization methods such as STRUM-LLM fall short of this goal. To overcome these limitations, we introduce Q-STRUM Debate, a novel extension of STRUM-LLM that employs debate-style prompting to generate focused and contrastive summarizations of item aspects relevant to a query. Leveraging modern large language models (LLMs) as powerful tools for generating debates, Q-STRUM Debate provides enhanced contrastive summaries. Experiments across three datasets demonstrate that Q-STRUM Debate yields significant performance improvements over existing methods on key contrastive summarization criteria, thus introducing a novel and performant debate prompting methodology for QCS.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Contrastive Summarization | Restaurants | Contrast87 | 2 | |
| Contrastive Summarization | Hotels | Contrast80 | 2 | |
| Contrastive Summarization | TravelDest | Contrast0.75 | 2 | |
| Pairwise LLM Evaluation | Restaurants | Win Rate (Contrast)83 | 2 | |
| Pairwise LLM Evaluation | Hotels | Win Rate (Contrast)82 | 2 | |
| Pairwise LLM Evaluation | TravelDest | Win Rate Contrast74 | 2 | |
| Query-driven contrastive summarization | Restaurants (test) | Contrast Win Rate87 | 2 | |
| Query-driven contrastive summarization | Hotels (test) | Contrast Win Rate82 | 2 | |
| Query-driven contrastive summarization | TravelDest (test) | Contrast Win Rate78 | 2 |