SCAN
Benchmarks
| Task Name | Dataset Name | SOTA Result | Trend | |
|---|---|---|---|---|
| Reward Modeling | SCAN HPD | Accuracy82.88 | 22 | |
| Instruction Following | SCAN jump | Accuracy100 | 18 | |
| Semantic Parsing | SCAN Around Right | Exact-match Accuracy100 | 16 | |
| Analogy generation | SCAN (out-of-domain) | Accuracy15.3 | 15 | |
| Systematic Generalization | SCAN Around Right (test) | Accuracy95.7 | 15 | |
| Systematic Generalization | SCAN Around Right (val) | Accuracy99.8 | 15 | |
| Systematic Generalization | SCAN Add Jump (test) | Accuracy99.8 | 15 | |
| Systematic Generalization | SCAN Add Jump (val) | Accuracy99.6 | 15 | |
| Language-driven Navigation | SCAN Simple v1.0 | Accuracy1 | 12 | |
| Semantic Parsing | SCAN MCD3 | Exact Match Accuracy80.2 | 12 | |
| Semantic Parsing | SCAN (MCD2) | Exact Match Accuracy80.8 | 12 | |
| Semantic Parsing | SCAN (MCD1) | Exact-match Accuracy0.674 | 12 | |
| Semantic Parsing | SCAN Jump | Exact-match Accuracy100 | 11 | |
| Command-to-action mapping | SCAN (length) | Accuracy99.7 | 11 | |
| Language-driven Navigation | SCAN around right v1.0 | Accuracy1 | 8 | |
| Instruction Following | SCAN around right | Accuracy99.51 | 7 | |
| Semantic Parsing | SCAN (MCD) | Accuracy100 | 6 | |
| Semantic Parsing | SCAN Template | Accuracy100 | 6 | |
| Semantic Parsing | SCAN (Length) | Accuracy100 | 6 | |
| Semantic Parsing | SCAN 0-shot lexical | Accuracy (0-shot)99 | 6 | |
| Semantic Parsing | SCAN 1-shot lexical | Accuracy100 | 6 | |
| Semantic Parsing | SCAN (IID) | Accuracy100 | 6 | |
| Around Right | SCAN (val) | Accuracy97.7 | 6 | |
| Add Jump | SCAN (val) | Accuracy96.9 | 6 | |
| Around Right | SCAN (test) | Accuracy77.9 | 6 |
Showing 25 of 34 rows