Type Inference on Private macOS Frameworks (MOTIF) (test)
75.2PM Accuracy
Qwen3 (8B)
Updated 4d ago
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
2026.01 | 75.2 | 48.4 | 100 | 0 | 100 | 0 | 60 | 40 | 0 | 33.3 | |
2026.01 | 72.5 | 38.7 | 100 | 38.7 | 97.3 | 2.7 | 60 | 20 | 0 | 16.7 | |
2026.01 | 71.9 | 51.6 | 100 | 0 | 100 | 0 | 60 | 0 | 0 | 33.3 | |
2026.01 | 67.7 | 32.3 | 100 | 9.7 | 100 | 0 | 60 | 20 | 0 | 50 | |
2026.01 | 66.7 | 56.2 | 100 | 0 | 100 | 0 | 33.3 | 0 | 0 | 50 | |
2026.01 | 58.3 | 48.4 | 100 | 6.5 | 100 | 0 | 40 | 0 | 0 | 0 | |
2026.01 | 37.3 | 6.5 | 0 | 6.5 | - | - | 60 | 0 | 0 | 0 | |
2026.01 | 24.6 | 6.5 | 83.9 | 0 | 100 | 0 | 40 | 0 | 0 | 20 | |
2026.01 | 22.3 | 0 | - | - | - | - | 100 | 0 | 0 | 0 |