Counterfactual Data Augmentation for Mitigating Gender Stereotypes in Languages with Rich Morphology
About
Gender stereotypes are manifest in most of the world's languages and are consequently propagated or amplified by NLP systems. Although research has focused on mitigating gender stereotypes in English, the approaches that are commonly employed produce ungrammatical sentences in morphologically rich languages. We present a novel approach for converting between masculine-inflected and feminine-inflected sentences in such languages. For Spanish and Hebrew, our approach achieves F1 scores of 82% and 73% at the level of tags and accuracies of 90% and 87% at the level of forms. By evaluating our approach using four different languages, we show that, on average, it reduces gender stereotyping by a factor of 2.5 without any sacrifice to grammaticality.
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Counterfactual Input Evaluation | CrowS-Pairs | SS55.35 | 33 | |
| Utility Evaluation | Anchor Utility Dataset | Anchor-PPL5.24 | 16 | |
| Safety Evaluation | Anchor Safety Dataset | Anchor Accuracy100 | 16 | |
| Mechanism Analysis | Model Internal Representations | Edge Delta Specification0.0499 | 16 | |
| Debiasing Effectiveness | In-Distribution (ID) | Mean Effectiveness Score (ID)1.14 | 16 | |
| Debiasing Effectiveness | Out-of-Distribution (OOD) Split | Mean Ratio1.26 | 16 | |
| Stereotype Bias Evaluation | StereoSet Gender | LMS Score85.47 | 15 | |
| Gender bias evaluation | SEAT | SEAT 60.596 | 13 | |
| Stereotypical Bias Evaluation | StereoSet (dev) | Overall LMS Score83.466 | 12 | |
| Bias Evaluation | HolisticBias | -- | 10 |