Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

MMEB

Benchmarks

Task NameDataset NameSOTA ResultTrend
Image EmbeddingMMEB v1 (test)
Classification69.1
70
Multimodal EmbeddingMMEB
Classification Accuracy76.1
56
Multi-modal EmbeddingMMEB 1.0 (test)
Classification Accuracy67.6
52
Multimodal RetrievalMMEB
Classification Score788.1
50
Multimodal Embedding EvaluationMMEB V2 (test)
Image CLS Hit@169.8
35
Multimodal Visual Document RetrievalMMEB Visual Document portion v2
ViDoRe V1 Score89.44
31
Multimodal Retrieval and UnderstandingMMEB V2 (test)
Image CLS Acc76.7
27
Multimodal RetrievalMMEB Image V2
CLS Accuracy69.1
22
Multimodal RankingMMEB
Classification Score70
22
Multimodal RetrievalMMEB v1 (test)
Classification61.2
18
Multi-modal Representation LearningMMEB OOD 1.0
OOD Precision@159.1
18
Multi-modal Representation LearningMMEB In-Distribution 1.0
MMEB IND Precision@171.6
18
Multi-modal Representation LearningMMEB Overall 1.0
Classification P@161.6
18
Multimodal Embedding EvaluationMMEB Overall
Classification Score72.6
18
RetrievalMMEB v2
Image Retrieval Score78.2
18
Video UnderstandingMMEB Video v2
Classification Score (CLS)57.8
17
Multimodal RetrievalMMEB Total v2
Overall Score68.1
15
Multimodal RetrievalMMEB Video V2
CLS Accuracy51.6
15
Image UnderstandingMMEB Image v2
Accuracy (CLS)68.1
9
Zero-shot Image ClassificationMMEB (val)
Image Classification Accuracy66.8
9
Multimodal Video RetrievalMMEB Video portion v2
K700 Score56.8
9
Video RetrievalMMEB Video Retrieval (MSRVTT, MSVD, DiDeMo, YouCook2, VATEX) v2 (test)
Retrieval Score43.1
8
Video ClassificationMMEB Video Classification (Kinetics-700, SSv2, HMDB, UCF, Breakfast) v2 (test)
Classification Accuracy63.7
8
Universal Multimodal EmbeddingMMEB Total v2
Total Score61.6
7
Video Question AnsweringMMEB Video QA v2 (test)
Average Score72.5
6
Showing 25 of 28 rows