Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

MMEB

Benchmarks

Task NameDataset NameSOTA ResultTrend
Multimodal RetrievalMMEB
Classification Score788.1
94
Multimodal EmbeddingMMEB
Classification Accuracy76.1
79
Image EmbeddingMMEB v1 (test)
Classification69.1
70
Multi-modal EmbeddingMMEB 1.0 (test)
Classification Accuracy67.6
52
Multimodal RetrievalMMEB Image V2
Overall Score78.2
37
Multimodal Retrieval and UnderstandingMMEB V2 (test)
Image CLS Acc76.7
37
Multimodal Embedding EvaluationMMEB V2 (test)
Image CLS Hit@169.8
35
Multimodal EmbeddingMMEB V1
Classification Accuracy69.7
31
Multimodal Visual Document RetrievalMMEB Visual Document portion v2
ViDoRe V1 Score89.44
31
Multimodal RankingMMEB
Classification Score70
22
Universal Multimodal RetrievalMMEB Full v2 (test)
Overall Average Score77.8
18
Multimodal Video RetrievalMMEB Video V2 (test)
CLS Hit@178.4
18
Multimodal RetrievalMMEB Image v2 (test)
CLS (Hit@1)76.1
18
Multimodal RetrievalMMEB v1 (test)
Classification61.2
18
Multi-modal Representation LearningMMEB OOD 1.0
OOD Precision@159.1
18
Multi-modal Representation LearningMMEB In-Distribution 1.0
MMEB IND Precision@171.6
18
Multi-modal Representation LearningMMEB Overall 1.0
Classification P@161.6
18
Multimodal Embedding EvaluationMMEB Overall
Classification Score72.6
18
RetrievalMMEB v2
Image Retrieval Score78.2
18
Video UnderstandingMMEB Video v2
Classification Score (CLS)57.8
17
Multimodal Retrieval and UnderstandingMMEB V2
Image CLS Accuracy70
16
Multimodal RerankingMMEB Summary v2 (Overall)
Overall Score79.2
15
Multimodal RerankingMMEB VisDoc v2
Overall Score87.1
15
Multimodal RerankingMMEB Video v2
Overall Average Score62.9
15
Multimodal RetrievalMMEB Total v2
Overall Score68.1
15
Showing 25 of 38 rows