Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

XMD: An End-to-End Framework for Interactive Explanation-Based Debugging of NLP Models

About

NLP models are susceptible to learning spurious biases (i.e., bugs) that work on some datasets but do not properly reflect the underlying task. Explanation-based model debugging aims to resolve spurious biases by showing human users explanations of model behavior, asking users to give feedback on the behavior, then using the feedback to update the model. While existing model debugging methods have shown promise, their prototype-level implementations provide limited practical utility. Thus, we propose XMD: the first open-source, end-to-end framework for explanation-based model debugging. Given task- or instance-level explanations, users can flexibly provide various forms of feedback via an intuitive, web-based UI. After receiving user feedback, XMD automatically updates the model in real time, by regularizing the model so that its explanations align with the user feedback. The new model can then be easily deployed into real-world applications via Hugging Face. Using XMD, we can improve the model's OOD performance on text classification tasks by up to 18%.

Dong-Ho Lee, Akshen Kadakia, Brihi Joshi, Aaron Chan, Ziyi Liu, Kiran Narahari, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, Xiang Ren• 2022

Related benchmarks

TaskDatasetResultRank
Sentiment AnalysisYelp Reviews (Out-of-domain)
Accuracy94.4
13
Hate Speech DetectionSTF (In-distribution)
Accuracy90
7
Hate Speech DetectionHatEval (Out-of-Distribution)
Accuracy90.1
7
Hate Speech DetectionGHC Out-of-Distribution
Accuracy67.9
7
Hate Speech DetectionLatent (Out-of-Distribution)
Accuracy70.3
7
Sentiment AnalysisSST In-distribution
Accuracy94.7
3
Sentiment AnalysisAmazon (Out-of-Distribution)
Accuracy92.3
3
Sentiment AnalysisMovies (Out-of-Distribution)
Accuracy94.5
3
Showing 8 of 8 rows

Other info

Code

Follow for update