Our new X account is live! Follow @wizwand_team for updates

Summarization Evaluation on SummEval

57Coherence

SMART-L

Updated 4d ago

Evaluation Results

Method	Links
SMART-L 2024.02		57	57	61	73	-
SMART-L 2022.12		56.7	56.7	61.1	73.3	61.9
BARTScore+CNN 2022.12		55	31.7	59.4	58.3	51.1
BLEURT 2022.12		53.3	20	41	46.7	40.3
RISE Multi-News 2022.12		53.3	73.3	71.1	70	66.9
RISE SamSUM 2022.12		53.3	70	67.8	70	65.3
RISE CNN 2022.12		53.3	73.3	74.5	70	67.8
BLEURT 2024.02		53	20	41	47	-
RISE extMulti-News 2024.02		53	73	71	70	-
RISE SamSUM 2024.02		53	70	68	70	-
RISE CNN 2024.02		53	73	75	70	-
SMART-1 2022.12		43.3	66.7	64.4	66.7	60.3
SMART-1 2024.02		43	67	64	67	-
SMART-2 2024.02		42	75	63	58	-
SMART-2 2022.12		41.7	75	62.8	58.3	59.4
ROUGE-1 2022.12		35	55	52.7	58.3	50.3
CHRF 2022.12		35	61.7	56.1	55	51.9
BARTScore 2022.12		35	61.7	49.4	45	47.8
ROUGE-1 2024.02		35	55	53	58	-
CHRF 2024.02		35	63	56	55	-
BARTScore 2024.02		35	62	49	45	-
BERTScore 2022.12		33.3	-3	14.2	20	16.1
BERTScore 2024.02		33	-3	14	20	-
BARTScore+CNN 2024.02		32	59	45	58	-
SMS 2024.02		27	60	36	40	-
SMS 2022.12		26.7	60	36	40	40.7
T5-ANLI 2022.12		25	58.3	54.4	51.7	47.3
Q2 2022.12		25	75	57.7	45	50.7
T5-ANLI 2024.02		25	58	54	52	-
Q2 2024.02		25	75	58	45	-
ROUGE-2 2022.12		23.3	60	49.4	43.3	44
PRISM 2022.12		23.3	60	36	36.7	39
ROUGE-2 2024.02		23	60	49	43	-
MoverScore 2024.02		23	-5	26	35	-
PRISM 2024.02		23	60	36	37	-
COSMIC (I(T; S)) 2024.02		23	53	47	54	-
BLEU 2024.02		22	5	33	38	-
BLEU 2022.12		21.7	5	32.6	38.3	24.4
MoverScore 2022.12		21.7	-5	25.9	35	19.4
ROUGE-L 2024.02		12	12	26	35	-
ROUGE-L 2022.12		11.7	11.7	25.9	35	21.1