Spider
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Text-to-SQL | Spider (test) | Execution Accuracy89.65 | 140 | |
| Text-to-SQL | Spider (dev) | EX (All)89.5 | 100 | |
| Text-to-SQL | Spider 1.0 (dev) | Exact Match Accuracy86.75 | 92 | |
| Text-to-SQL | Spider 1.0 (test) | EM Acc (Overall)87.2 | 91 | |
| Text-to-SQL | Spider | Exec Acc (All)86.54 | 57 | |
| Text2SQL | Spider (test) | Exec Acc (Greedy)88.3 | 37 | |
| Text-to-SQL | Spider-Realistic | Execution Accuracy (EX)81.9 | 33 | |
| Text-to-SQL | Spider-Syn | Execution Accuracy (EX)78.7 | 26 | |
| Text-to-SQL | Spider-DK | Execution Accuracy (EX)77.8 | 26 | |
| SQL Semantic Validation | Spider | AUPRC63.21 | 24 | |
| Semantic Validation | Spider 2.0 | AUPRC92.59 | 18 | |
| SQL execution performance | Spider n=1034 | EM (1 Table)84.3 | 18 | |
| Text-to-SQL | Spider 1.0 (val) | Accuracy (All)64.8 | 11 | |
| Semantic Parsing | Spider (dev) | Exact Match Accuracy75.5 | 11 | |
| Text-to-SQL | Spider-Syn (dev) | Exact Match Accuracy62.6 | 11 | |
| Table Selection | SPIDER 2018 (test) | Avg Tables2.7 | 10 | |
| Text-to-SQL | Spider hidden (test) | Exact Match (EM)72.1 | 10 | |
| Text-to-SQL | Spider-SYN 1.0 (val) | EM Accuracy66.9 | 10 | |
| Text-to-SQL | Spider ADVETA-ADD | Exact Match (EM)50.6 | 10 | |
| Text-to-SQL | Spider (ADVETA-RPL) | Exact Match (EM)35.8 | 10 | |
| Text-to-SQL | Spider SmBoP parser outputs (dev) | EM78 | 9 | |
| Text-to-SQL | Spider BRIDGEv2 parser outputs (dev) | EM72.5 | 9 | |
| Text-to-SQL | Spider CodeT5 parser outputs (dev) | EM69.2 | 9 | |
| Text-to-SQL | Spider-Realistic 1.0 (test) | Exact Match (EM)77.4 | 9 | |
| Text-to-SQL (Component Matching) | Spider (test) | SELECT Component Error12 | 9 |
Showing 25 of 41 rows