Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining

About

While online conversations can cover a vast amount of information in many different formats, abstractive text summarization has primarily focused on modeling solely news articles. This research gap is due, in part, to the lack of standardized datasets for summarizing online discussions. To address this gap, we design annotation protocols motivated by an issues--viewpoints--assertions framework to crowdsource four new datasets on diverse online conversation forms of news comments, discussion forums, community question answering forums, and email threads. We benchmark state-of-the-art models on our datasets and analyze characteristics associated with the data. To create a comprehensive benchmark, we also evaluate these models on widely-used conversation summarization datasets to establish strong baselines in this domain. Furthermore, we incorporate argument mining through graph construction to directly model the issues, viewpoints, and assertions present in a conversation and filter noisy input, showing comparable or improved results according to automatic and human evaluations.

Alexander R. Fabbri, Faiaz Rahman, Imad Rizvi, Borui Wang, Haoran Li, Yashar Mehdad, Dragomir Radev• 2021

Related benchmarks

TaskDatasetResultRank
Abstractive SummarizationSamSum--
73
Meeting SummarizationICSI manual transcriptions--
22
Abstractive SummarizationStack ConvoSumm 1.0 (test)
ROUGE-139.73
11
Abstractive SummarizationNYT ConvoSumm 1.0 (test)
ROUGE-136.6
5
Abstractive SummarizationReddit ConvoSumm 1.0 (test)
ROUGE-136.51
5
Abstractive SummarizationConvoSumm Email 1.0 (test)
ROUGE-140.32
5
Meeting SummarizationAMI (test)
ROUGE-154.47
4
Abstractive Conversation SummarizationReddit (randomly selected 25 examples)
Relevance3.47
2
Abstractive Conversation SummarizationAMI randomly selected 10 examples
Relevance4.13
2
Abstractive SummarizationCQASUMM
ROUGE-132.79
2
Showing 10 of 13 rows

Other info

Code

Follow for update