NumNet: Machine Reading Comprehension with Numerical Reasoning
About
Numerical reasoning, such as addition, subtraction, sorting and counting is a critical skill in human's reading comprehension, which has not been well considered in existing machine reading comprehension (MRC) systems. To address this issue, we propose a numerical MRC model named as NumNet, which utilizes a numerically-aware graph neural network to consider the comparing information and performs numerical reasoning over numbers in the question and passage. Our system achieves an EM-score of 64.56% on the DROP dataset, outperforming all existing machine reading comprehension models by considering the numerical relations among numbers.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu• 2019
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Reading Comprehension | DROP (dev) | F1 Score85.59 | 63 | |
| Reading Comprehension | DROP (test) | F1 Score86.16 | 61 | |
| Numerical Question Answering | FinQA (test) | Execution Accuracy10.29 | 33 | |
| Numerical Question Answering | FinQA 1.0 (test) | Execution Accuracy48.57 | 14 | |
| Question Answering | MultiHiertt (test) | EM10.77 | 11 | |
| Question Answering | DROP (dev) | EM81.1 | 10 | |
| Question Answering | MULTIHIERTT (dev) | EM10.32 | 10 | |
| Numerical Reasoning Question Answering | FinQA v1 (dev) | Execution Accuracy47.53 | 7 | |
| Question Answering | TAT-QA 1.0 (test) | EM37 | 6 | |
| Question Answering | TAT-QA 1.0 (dev) | EM38.1 | 5 |
Showing 10 of 12 rows