Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

About

We present an audio-visual multimodal approach for the task of zeroshot learning (ZSL) for classification and retrieval of videos. ZSL has been studied extensively in the recent past but has primarily been limited to visual modality and to images. We demonstrate that both audio and visual modalities are important for ZSL for videos. Since a dataset to study the task is currently not available, we also construct an appropriate multimodal dataset with 33 classes containing 156,416 videos, from an existing large scale audio event dataset. We empirically show that the performance improves by adding audio modality for both tasks of zeroshot classification and retrieval, when using multimodal extensions of embedding learning methods. We also propose a novel method to predict the `dominant' modality using a jointly learned modality attention network. We learn the attention in a semi-supervised setting and thus do not require any additional explicit labelling for the modalities. We provide qualitative validation of the modality specific attention, which also successfully generalizes to unseen test classes.

Kranti Kumar Parida, Neeraj Matiyali, Tanaya Guha, Gaurav Sharma• 2019

Related benchmarks

TaskDatasetResultRank
Generalized Zero-Shot LearningCUB
H Score301
250
Generalized Zero-Shot LearningSUN
H26.6
184
Generalized Zero-Shot LearningAWA2
S Score36.8
165
Generalized Zero-Shot LearningAWA1
S Score47.9
49
Audio-visual Zero-Shot ClassificationVGGSound GZSL (test)
S Score10.86
38
Generalized Zero-Shot Retrieval (Text-to-Audio)AudioSet ZSL (test)
mAP (S)65.45
19
Audio-visual Zero-Shot ClassificationUCF GZSL cls (test)
S (Seen Accuracy)33.89
19
Audio-visual Zero-Shot ClassificationActivityNet GZSL cls (test)
S (Seen)10.75
19
Audio-visual Zero-Shot LearningActivityNet GZSL
S Score5.55
17
Audio-visual Zero-Shot LearningUCF GZSL
S (Seen Accuracy)26.04
17
Showing 10 of 24 rows

Other info

Code

Follow for update