Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

MMEB

Benchmarks

Task NameDataset NameSOTA ResultTrend
Multimodal RetrievalMMEB
Classification Score788.1
94
Image EmbeddingMMEB v1 (test)
Classification69.1
70
Multimodal EmbeddingMMEB
Classification Accuracy76.1
66
Multi-modal EmbeddingMMEB 1.0 (test)
Classification Accuracy67.6
52
Multimodal Retrieval and UnderstandingMMEB V2 (test)
Image CLS Acc76.7
37
Multimodal Embedding EvaluationMMEB V2 (test)
Image CLS Hit@169.8
35
Multimodal Visual Document RetrievalMMEB Visual Document portion v2
ViDoRe V1 Score89.44
31
Multimodal RetrievalMMEB Image V2
CLS Accuracy69.1
22
Multimodal RankingMMEB
Classification Score70
22
Universal Multimodal RetrievalMMEB Full v2 (test)
Overall Average Score77.8
18
Multimodal Video RetrievalMMEB Video V2 (test)
CLS Hit@178.4
18
Multimodal RetrievalMMEB Image v2 (test)
CLS (Hit@1)76.1
18
Multimodal RetrievalMMEB v1 (test)
Classification61.2
18
Multi-modal Representation LearningMMEB OOD 1.0
OOD Precision@159.1
18
Multi-modal Representation LearningMMEB In-Distribution 1.0
MMEB IND Precision@171.6
18
Multi-modal Representation LearningMMEB Overall 1.0
Classification P@161.6
18
Multimodal Embedding EvaluationMMEB Overall
Classification Score72.6
18
RetrievalMMEB v2
Image Retrieval Score78.2
18
Multimodal EmbeddingMMEB V1
Classification Accuracy68.3
17
Video UnderstandingMMEB Video v2
Classification Score (CLS)57.8
17
Multimodal RetrievalMMEB Total v2
Overall Score68.1
15
Multimodal RetrievalMMEB Video V2
CLS Accuracy51.6
15
Multimodal Dense RetrievalMMEB V2
Image Recall@180.15
10
Video RetrievalMMEB Video Retrieval (MSRVTT, MSVD, DiDeMo, YouCook2, VATEX) v2
Retrieval Score39.3
10
Video ClassificationMMEB Kinetics-700, SSv2, HMDB, UCF, Breakfast v2
Classification Accuracy63.2
10
Showing 25 of 34 rows