Prediction: AI will make formal verification go mainstream
Martin Kleppmann predicts that AI will elevate formal verification from an academic niche to a mainstream software development practice. This transformation is driven by AI's ability to dramatically reduce the cost and complexity of generating mathematical proofs for code, making rigorous verification economically viable. The confluence of cheaper proofs and the imperative to verify AI-generated code will fundamentally reshape how we ensure software correctness.
The Lowdown
Martin Kleppmann posits that Artificial Intelligence is poised to usher formal verification, long considered a niche academic discipline, into the mainstream of software engineering. He argues that AI's capabilities will fundamentally alter the economics and practicality of ensuring software correctness through mathematical proofs.
- Formal Verification Explained: Traditional formal verification uses proof assistants (e.g., Rocq, Isabelle, Lean) to mathematically prove that code satisfies a formal specification, covering even edge cases that manual testing might miss.
- Current Obstacles: Historically, formal verification has been expensive and laborious, requiring PhD-level expertise. For instance, verifying seL4's microkernel required 20 person-years and 200,000 lines of Isabelle code for just 8,700 lines of C.
- AI's Impact on Cost: LLMs are becoming adept at writing proof scripts, drastically lowering the effort and cost associated with generating proofs, thus changing the economic equation for formal verification.
- New Imperative for Verification: AI-generated code introduces a new need for verification; rather than human review, AI can prove its own code is correct, a preferable alternative to "handcrafted code with artisanal bugs."
- Hallucination Mitigation: LLMs' probabilistic nature is counteracted by the deterministic proof checker, which rejects invalid proofs, forcing the AI to retry until a correct proof is found.
- Shifting Challenge: While proof generation becomes automated, the new frontier will be accurately defining formal specifications, a task that still requires human expertise but is less laborious than writing proofs.
- Future Vision: Kleppmann envisions a future where developers declaratively specify code properties, and AI generates both the implementation and its formal proof, similar to how compilers generate machine code without manual inspection.
In essence, the confluence of formal verification becoming vastly cheaper, the inherent need to verify AI-generated code, and the precision of verification countering LLM imprecision will drive formal methods mainstream. The primary hurdle remaining will be the cultural adoption of these newly viable practices.