Symbolic Reasoning

Benchmarks

Dataset Name	SOTA Method	Metric
Last Letter Concatenation	Zero-Shot CoT	Accuracy90.4	68	20d ago
Letter	Meta-Reasoning Paradigm	Accuracy92.4	67	2mo ago
Coin	Meta-Reasoning Paradigm	Accuracy100	45	2mo ago
Last Letter	In-Writing-BF	Accuracy0.819	31	20d ago
Coin Flip	Auto-CoT	Accuracy99.9	27	2mo ago
AQUA	COT	Accuracy80.3	26	3mo ago
OlyBench	QuaSAR	Accuracy36.5	25	3mo ago
MMLU Redux	QuaSAR	Accuracy80.3	25	3mo ago
Countdown	FDM-A	Accuracy49.61	24	3mo ago
Date	Automatic Model Selection with LLMs	Solve Rate90.5	14	3mo ago
Lies	Meta-Reasoning Paradigm	Accuracy100	12	3mo ago
LastLetter (test)	SGE	Accuracy86.96	11	3mo ago
Coin Flip	RM-Primed	Accuracy80.75	10	2mo ago
Coinflip 4	Self-consistency	Accuracy99.5	10	3mo ago
Date Understanding (DU)	SoftCoT	Accuracy87.2	10	3mo ago
CoinFlip	SC+IC (tune)	Calibrated Accuracy100	8	3mo ago
Last Letter Concatenation (test)		Accuracy81.9	8	3mo ago
Last Letter Concatenation (LLC) out-of-distribution (test)	StrategyLLM	LLC-4 Accuracy98	7	3mo ago
COLOR (Colored Object)	SATLM	Accuracy99.4	7	3mo ago
ARC-AGI 1 (test)	RIMA	Pass@247.5	6	2mo ago
Coin Flip OOD (test)	AMPLIFY	Accuracy65.7	6	3mo ago
Coin Flip (test)	Few-shot Auto-CoT	Accuracy98.6	6	3mo ago
Symbolic Longer	SCO	Accuracy (Clean, Avg)0.187	5	3mo ago
Symbolic Equal	CD-CoT	Acc (Clean, Avg)42.7	5	3mo ago
Date Understanding (DU) (test)	SoftCoT	Accuracy67.52	4	3mo ago

Showing 25 of 43 rows