Share your thoughts, 1 month free Claude Pro on usSee more
WorkDL logo mark

POLAR: A Benchmark for Multilingual, Multicultural, and Multi-Event Online Polarization

About

Online polarization poses a growing challenge for democratic discourse, yet most computational social science research remains monolingual, culturally narrow, or event-specific. We introduce POLAR, a multilingual, multicultural, and multi-event dataset with over 110K instances in 22 languages drawn from diverse online platforms and real-world events. Polarization is annotated along three axes, namely detection, type, and manifestation, using a variety of annotation platforms adapted to each cultural context. We conduct two main experiments: (1) fine-tuning six pretrained small language models; and (2) evaluating a range of open and closed large language models in few-shot and zero-shot settings. The results show that, while most models perform well in binary polarization detection, they achieve substantially lower performance when predicting polarization types and manifestations. These findings highlight the complex, highly contextual nature of polarization and demonstrate the need for robust, adaptable approaches in NLP and computational social science. All resources will be released to support further research and effective mitigation of digital polarization globally.

Usman Naseem, Robert Geislinger, Juan Ren, Sarah Kohail, Rudy Garrido Veliz, P Sam Sahil, Yiran Zhang, Marco Antonio Stranisci, Idris Abdulmumin, \"Ozge Alacam, Cengiz Acart\"urk, Aisha Jabr, Saba Anwar, Abinew Ali Ayele, Simona Frenda, Alessandra Teresa Cignarella, Elena Tutubalina, Oleg Rogov, Aung Kyaw Htet, Xintong Wang, Surendrabikram Thapa, Kritesh Rauniyar, Tanmoy Chakraborty, Arfeen Zeeshan, Dheeraj Kodati, Satya Keerthi, Sahar Moradizeyveh, Firoj Alam, Arid Hasan, Syed Ishtiaque Ahmed, Ye Kyaw Thu, Shantipriya Parida, Ihsan Ayyub Qazi, Lilian Wanzare, Nelson Odhiambo Onyango, Clemencia Siro, Jane Wanjiru Kimani, Ibrahim Said Ahmad, Adem Chanie Ali, Martin Semmann, Chris Biemann, Shamsuddeen Hassan Muhammad, Seid Muhie Yimam• 2025

Related benchmarks

TaskDatasetResultRank
Multilingual Taxonomy EnrichmentSemEval Subtask 1
F1 Score87.98
44
Binary Polarization DetectionSubtask 1 English (test)
Macro F178.02
3
Binary Polarization DetectionSubtask 1 Swahili (test)
Macro F175.71
3
Multi-label Type ClassificationSubtask 2 Swahili (test)
Macro F144.17
3
Multi-label Manifestation ClassificationSubtask 3 English (test)
Macro F141
3
Multi-label Manifestation ClassificationSubtask 3 Swahili (test)
Macro F122.05
3
Multi-label Type ClassificationSubtask 2 English (test)
Macro F133.33
3
Subtask 2SemEval Subtask 2 (hin)
F1 Score79.11
2
Subtask 2SemEval Subtask 2 ita
F1 Score37.59
2
Subtask 2SemEval Subtask 2 (ori)
F1 Score56
2
Showing 10 of 30 rows

Other info

Follow for update