On the Unreasonable Effectiveness of Property-Based Testing for Validating Formal Specifications
Summary
The post argues that property-based testing (PBT) can effectively validate LLM-synthesised specifications for Lean programs, offering a cheaper alternative to symbolic proofs. It introduces three checks—Admissibility, Soundness, and Uniqueness—for validating specifications, discusses the limitations of SPOTs in Lean, and presents a PBT-based pipeline used to uncover underspecified specs in VERINA and CLEVER benchmarks, with several concrete examples and future directions.