SOTA Symbolic Reasoning benchmarks and papers with code

Benchmarks

Dataset Name	SOTA Method	Metric
Last Letter Concatenation	Zero-Shot CoT	Accuracy90.4	46	2d ago
Letter	Meta-Reasoning Paradigm	Accuracy92.4	33	2d ago
AQUA	COT	Accuracy80.3	26	4d ago
OlyBench	QuaSAR	Accuracy36.5	25	4d ago
MMLU Redux	QuaSAR	Accuracy80.3	25	4d ago
Countdown	FDM-A	Accuracy49.61	24	4d ago
Last Letter	In-Writing-BF	Accuracy0.819	21	4d ago
Coin Flip	Auto-CoT	Accuracy99.9	15	2d ago
Date	Automatic Model Selection with LLMs	Solve Rate90.5	14	4d ago
Lies	Meta-Reasoning Paradigm	Accuracy100	12	2d ago
LastLetter (test)	SGE	Accuracy86.96	11	4d ago
Coin	Meta-Reasoning Paradigm	Accuracy100	11	2d ago
Coinflip 4	Self-consistency	Accuracy99.5	10	2d ago
Date Understanding (DU)	SoftCoT	Accuracy87.2	10	4d ago
CoinFlip	SC+IC (tune)	Calibrated Accuracy100	8	4d ago
Last Letter Concatenation (test)		Accuracy81.9	8	4d ago
Last Letter Concatenation (LLC) out-of-distribution (test)	StrategyLLM	LLC-4 Accuracy98	7	2d ago
COLOR (Colored Object)	SATLM	Accuracy99.4	7	4d ago
Coin Flip OOD (test)	AMPLIFY	Accuracy65.7	6	4d ago
Coin Flip (test)	Few-shot Auto-CoT	Accuracy98.6	6	4d ago
Symbolic Longer	SCO	Accuracy (Clean, Avg)0.187	5	4d ago
Symbolic Equal	CD-CoT	Acc (Clean, Avg)42.7	5	4d ago
Date Understanding (DU) (test)	SoftCoT	Accuracy67.52	4	4d ago
Penguins	PAL Codex	Solve Rate93.3	4	4d ago
Coinflip OOD: 4	PaLM 540B	Accuracy90.2	3	2d ago

Showing 25 of 36 rows