Share your thoughts, 1 month free Claude Pro on usSee more

Multimodal Science Question Answering on ScienceQA v1.0 (test)

97.47Accuracy (Natural Language Component)

MC-CoT_F-Large

Updated 1mo ago

Evaluation Results

Method
MC-CoT_F-Large 2023.11	97.47	90.44	93.18	96.97	93.75	94.49	95.3	94.13	94.88
Mutimodal-CoT_Large 2023.11	95.91	82	90.82	95.26	88.8	92.89	92.44	90.31	91.68
MC-CoT_Large 2023.11	95.47	89.99	91.82	95.11	92.66	93.24	94.27	91.76	93.37
MC-CoT_F-Base 2023.11	93.56	83.58	90.73	94.13	89.24	90.94	90.93	90.38	90.73
MC-CoT_Base 2023.11	91.87	84.59	93	92.28	88.3	92.75	90.64	90.64	90.64
LLaVa + GPT-4 2023.11	91.56	96.74	91.09	90.62	88.99	93.52	92.73	92.16	92.53
LLaVa 2023.11	90.36	95.95	88	89.49	88	90.66	90.93	90.9	90.92
LaVIN-13B 2023.11	90.32	94.38	87.73	89.44	87.65	90.31	91.19	89.26	90.5
Human 2023.11	90.23	84.97	87.48	89.6	87.5	88.1	91.59	82.42	88.4
Chameleon + GPT-4 2023.11	89.83	74.13	89.82	88.27	77.64	92.13	88.03	83.72	86.54
LLaMA-SciTune_Large 2023.11	89.3	95.61	87	93.08	86.67	91.75	84.37	91.3	90.03
LaVIN-7B 2023.11	89.25	94.94	85.24	88.51	87.46	88.08	90.16	88.07	89.41
Mutimodal-CoT_Base 2023.11	87.52	77.17	85.82	87.88	82.9	86.83	84.65	85.37	84.91
GPT-4 w/ CoT 2023.11	85.48	72.44	90.27	82.65	71.49	92.89	86.66	79.04	83.99
LLaMA-SciTune_Base 2023.11	84.5	94.15	82.91	88.35	83.64	88.74	85.05	85.6	86.11
LLaMA-Adapter 2023.11	84.37	88.3	84.36	83.72	80.32	86.9	85.83	84.05	85.19
Chameleon + ChatGPT 2023.11	81.62	70.64	84	79.77	70.8	86.62	81.86	76.53	79.93
LLaMA-Adapter (T) 2023.11	79	73.79	80.55	78.3	70.35	83.14	79.77	75.68	78.31
ChatGPT w/ CoT 2023.11	78.82	70.98	83.18	77.37	67.92	86.13	80.72	74.03	78.31
GPT-3.5 w/ CoT 2023.11	75.44	70.87	78.09	74.68	67.43	79.93	78.23	69.68	75.17
GPT-3.5 2023.11	74.64	69.74	76	74.44	67.28	77.42	76.8	68.89	73.97
UnifiedQA_Base w/ COT 2023.11	71	76.04	78.91	66.42	66.53	81.81	77.06	68.82	74.11
UnifiedQA_Base 2023.11	68.16	69.18	74.91	63.78	61.38	77.84	72.98	65	70.12
Patch-TRM 2023.11	65.19	46.79	65.55	66.96	55.28	64.95	58.04	67.5	61.42
DFAF 2023.11	64.03	48.82	63.55	65.88	54.49	64.11	57.12	67.17	60.72
BAN 2023.11	60.88	46.57	66.64	62.61	52.6	65.51	56.83	63.94	59.37
VILT 2023.11	60.48	63.89	60.27	63.2	61.38	57	60.72	61.9	61.14
Top-Down 2023.11	59.5	54.33	61.82	62.9	54.88	59.79	57.27	62.16	59.02
VisualBERT 2023.11	59.33	69.18	61.18	62.71	62.17	58.54	62.96	59.92	61.87
MCAN 2023.11	56.08	46.23	58.09	59.43	51.17	55.4	51.65	59.72	54.54
Random Choice 2023.11	40.28	46.13	29.25	47.75	40.08	33.66	39.35	40.67	39.83