Share your thoughts, 1 month free Claude Pro on usSee more

LLM Judge Consistency Evaluation on RewardBench

83.25Kappa

CalibraEval

Updated 1mo ago

Evaluation Results

Method	Links
CalibraEval 2024.10		83.25	96.25	96.27
CalibraEval 2024.10		82.8	95.47	95.75
GPT4o (Default) 2024.10		82.57	94.83	94.89
Pride 2024.10		82.53	94.94	94.98
CC 2024.10		81.38	94.47	94.48
Pride 2024.10		78.64	92.99	93.3
DI 2024.10		78.57	93.72	94.01
Qwen-72B (Default) 2024.10		78.28	92.77	93.13
DI 2024.10		77.33	92.27	92.63
DC 2024.10		76.89	92.28	92.35
CC 2024.10		69.23	88.71	89.92
DC 2024.10		66.61	87.03	87.9
CalibraEval 2024.10		32.02	77.25	77.6
CalibraEval 2024.10		30.32	86.51	86.66
Pride 2024.10		28.25	70.38	73.16
DC 2024.10		27.94	66.09	70.54
Pride 2024.10		27.65	72.72	73.77
CalibraEval 2024.10		26.4	75.75	76.11
Pride 2024.10		24.73	67.46	68.3
DI 2024.10		24.52	70.23	71.58
CC 2024.10		24.28	57.25	58.23
DC 2024.10		23.57	69.04	72.79
DC 2024.10		23.17	72.89	76.22
DI 2024.10		21.59	74.04	80
Llama-3-8B (Default) 2024.10		20.81	66.24	71.79
CalibraEval 2024.10		20.67	83.23	86.68
ChatGPT (Default) 2024.10		20.08	62.67	70.75
Qwen-14B (Default) 2024.10		19.69	63.41	66.86
DI 2024.10		19.63	64.98	70.87
CC 2024.10		16.89	59.24	60.09
DC 2024.10		16.04	45.1	45.26
CC 2024.10		15.49	58.77	63.7
Llama-3.1-8B (Default) 2024.10		15.02	68.82	76.65
Pride 2024.10		14.63	66.62	76.02
DI 2024.10		11.9	50.46	53.2
CC 2024.10		-1.62	-4.71	-4.93