LLMSTEP: LLM proofstep suggestions in Lean
About
We present LLMSTEP, a tool for integrating a language model into the Lean proof assistant. LLMSTEP is a Lean 4 tactic that sends a user's proof state to a server hosting a language model. The language model generates suggestions, which are checked in Lean and displayed to a user in their development environment. We provide a baseline language model, along with code for fine-tuning and evaluation to support further development. We provide server implementations that run on CPU, a CUDA GPU, or a Google Colab notebook, as a step towards fast, effective language model suggestions for any user.
Sean Welleck, Rahul Saha• 2023
Related benchmarks
| Task | Dataset | Result | Rank | |
|---|---|---|---|---|
| Formal Theorem Proving | MiniF2F (test) | Pass@129.51 | 100 | |
| Automated Theorem Proving | MiniF2F (test) | Success Rate23.36 | 93 | |
| Theorem Proving | miniF2F (val) | Success Rate31.56 | 59 | |
| Theorem Proving | ProofNet (test) | Pass Rate (%)13.56 | 12 | |
| Theorem Proving | ProofNet (val) | Accuracy10.17 | 11 | |
| Formal Theorem Proving | ProofNet (val) | Pass Rate8.47 | 6 |
Showing 6 of 6 rows