Our new X account is live! Follow @wizwand_team for updates
WorkDL logo mark

LLMSTEP: LLM proofstep suggestions in Lean

About

We present LLMSTEP, a tool for integrating a language model into the Lean proof assistant. LLMSTEP is a Lean 4 tactic that sends a user's proof state to a server hosting a language model. The language model generates suggestions, which are checked in Lean and displayed to a user in their development environment. We provide a baseline language model, along with code for fine-tuning and evaluation to support further development. We provide server implementations that run on CPU, a CUDA GPU, or a Google Colab notebook, as a step towards fast, effective language model suggestions for any user.

Sean Welleck, Rahul Saha• 2023

Related benchmarks

TaskDatasetResultRank
Formal Theorem ProvingMiniF2F (test)
Pass@129.51
100
Automated Theorem ProvingMiniF2F (test)
Success Rate23.36
93
Theorem ProvingminiF2F (val)
Success Rate31.56
59
Theorem ProvingProofNet (test)
Pass Rate (%)13.56
12
Theorem ProvingProofNet (val)
Accuracy10.17
11
Formal Theorem ProvingProofNet (val)
Pass Rate8.47
6
Showing 6 of 6 rows

Other info

Follow for update