Share your thoughts, 1 month free Claude Pro on usSee more

Zero-shot Reasoning

Benchmarks

Dataset Name	SOTA Method	Metric
Reasoning Suite Zero-shot (PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c) (val test)		PIQA85.56	177	2d ago
PIQA		PIQA Zero-shot Accuracy80.9	62	9d ago
Evaluation Suite Zero-shot (OpenbookQA, ARC-e, ARC-c, WinoGrande, HellaSwag, PIQA, MathQA)	Meta-Llama-3-8B Dense	Average Accuracy69.99	56	1mo ago
Reasoning Tasks (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, OBQA) Zero-shot		BoolQ Accuracy (Zero-shot)82.813	55	29d ago
Zero-Shot Reasoning Tasks (ARC-C, ARC-E, BoolQ, Hella, OBQA, PIQA, SIQA, Wino)		ARC-C Accuracy65.53	54	2d ago
WinoGrande		Accuracy70	54	9d ago
HellaSwag		Accuracy76.3	48	15d ago
Reasoning Suite (ARC-e, ARC-c, HellaSwag, PIQA, Winogrande) zero-shot		ARC-e Accuracy81.06	41	1mo ago
ARC-Easy zero-shot	WeDLM-8B	Zero-shot Accuracy97.43	41	15d ago
ARC-e, Winogrande, HellaSwag, PIQA		Normalized Avg Accuracy77.2	36	1mo ago
MathQA		Accuracy28.4	26	15d ago
OpenbookQA		Accuracy44	26	15d ago
benchmark datasets (PIQA, HeSw, ARC-e, ARC-c, OBQA, Race, WSC273, LAMBADA, MMLU) Zero-shot	LLaMA-2-7B-hf	PIQA78.07	21	1mo ago
Reasoning Suite PiQA, LAMBDA, ARC, HellaSwag	NSA	PiQA Score62.69	20	24d ago
ARC-e, PIQA, OpenbookQA, Winogrande, HellaSwag, MathQA		Average Accuracy57	19	15d ago
Reasoning Tasks (PIQA, ARC-e, HS, WG) Zero-shot	Llama2-13B (W16A16)	PIQA Accuracy80.41	18	22d ago
ZeroShot 7		Accuracy67	16	1mo ago
Multiple Reasoning Datasets Combined	PLDRv51-SOC-110M-5	Average Score 042.62	11	23d ago
Zero-shot Average		Accuracy66	11	1mo ago
Reasoning Suite Zero-shot (PIQA, ARC, HS, WG, BoolQ, MMLU)	SliderQuant	PIQA Accuracy80.2	9	22d ago
StoryCloze zero-shot		Accuracy79.95	8	1mo ago

Showing 21 of 21 rows