BERTweet: A pre-trained language model for English Tweets
About
We present BERTweet, the first public large-scale pre-trained language model for English Tweets. Our BERTweet, having the same architecture as BERT-base (Devlin et al., 2019), is trained using the RoBERTa pre-training procedure (Liu et al., 2019). Experiments show that BERTweet outperforms strong baselines RoBERTa-base and XLM-R-base (Conneau et al., 2020), producing better performance results than the previous state-of-the-art models on three Tweet NLP tasks: Part-of-speech tagging, Named-entity recognition and text classification. We release BERTweet under the MIT License to facilitate future research and applications on Tweet data. Our BERTweet is available at https://github.com/VinAIResearch/BERTweet
Dat Quoc Nguyen, Thanh Vu, Anh Tuan Nguyen• 2020
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Semantic Textual Similarity | STS tasks (STS12, STS13, STS14, STS15, STS16, STS-B, SICK-R) various (test) | STS12 Score29.2 | 412 | |
| Named Entity Recognition | Wnut 2017 | F1 Score56.5 | 91 | |
| Named Entity Recognition | WNUT 2017 (test) | F1 Score57.1 | 63 | |
| Text Classification | TweetEVAL (test) | Accuracy (A)83.95 | 44 | |
| Multi-label Text Classification | Hotel Reviews (HR) (test) | F-Measure80.1 | 44 | |
| Part-of-Speech Tagging | TWEEBANK V2 (test) | Accuracy95.38 | 38 | |
| Named Entity Recognition | WNUT 2016 | F1 Score52.1 | 38 | |
| Named Entity Recognition | WNUT 2016 (test) | F1 Score52.1 | 26 | |
| Transfer Learning | SentEval Transfer tasks (test) | MR79.58 | 23 | |
| POS Tagging | Ritter11 T-POS (test) | Accuracy90.1 | 20 |
Showing 10 of 29 rows