Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

MLLMU-Bench

Benchmarks

Task NameDataset NameSOTA ResultTrend
ClassificationMLLMU-Bench Forget Set
Accuracy63.33
51
Visual Question Answering (VQA)MLLMU-Bench 5% (forget)
Accuracy (Classification)55
42
GenerationMLLMU-Bench (Forget Set)
Rouge Score64.5
37
Multimodal Machine Unlearning EvaluationMLLMU-Bench Forget Set
Classification Accuracy54.67
36
ClassificationMLLMU-Bench (Retain Set)
Accuracy67.07
32
ClassificationMLLMU-Bench (test)
Accuracy52.5
32
ClozeMLLMU-Bench (Forget Set)
Cloze Accuracy26.09
32
MLLM UnlearningMLLMU-Bench Retain Set 10% ratio
Cloze Accuracy34.4
30
MLLM UnlearningMLLMU-Bench 10% ratio (test)
Cloze Accuracy40
30
MLLM UnlearningMLLMU-Bench forget set, 10% ratio
Cloze Accuracy40
30
Multimodal Language Model UnlearningMLLMU-Bench 1.0 (Retain Set)
Cloze Accuracy30
30
Multimodal Language Model UnlearningMLLMU-Bench 1.0 (test)
Cloze Accuracy38.37
30
Multimodal Language Model UnlearningMLLMU-Bench forget set 1.0
Cloze Accuracy35.14
30
Open-Ended GenerationMLLMU-Bench (Retain Set)
ROUGE-L53.1
30
Cloze TaskMLLMU-Bench (Retain Set)
Accuracy24.52
30
Open-Ended GenerationMLLMU-Bench (test)
ROUGE-L34.5
30
Cloze TaskMLLMU-Bench (test)
Accuracy13.04
30
Multimodal Machine Unlearning EvaluationMLLMU-Bench Real Celebrity
Class Acc56.41
28
Multimodal Machine Unlearning EvaluationMLLMU-Bench (test)
Classification Accuracy47.86
27
Multimodal Machine UnlearningMLLMU-Bench LLaVA-1.5-7B (test 2)
Forget Rate62.8
24
Multimodal Machine UnlearningMLLMU-Bench LLaVA-1.5-7B (test 1)
Forget Rate65.4
24
Visual Question Answering (VQA)MLLMU-Bench 5% forget (Real)
Classification Accuracy78.59
21
Visual Question Answering (VQA)MLLMU-Bench 5% forget (test)
Classification Accuracy62.5
21
Visual Question Answering (VQA)MLLMU-Bench 5% forget
Contextual Refusal Rate0.01
18
Multimodal Machine UnlearningMLLMU-Bench
Forget VQA Accuracy29.6
16
Showing 25 of 34 rows