Many SWE-bench-Passing PRs Would Not Be Merged into Main
Summary
A METR study compares SWE-bench automated grading vs maintainer review for AI-generated patches. It finds a ~24-point gap and cautions against over-reliance on benchmarks; discusses robustness checks and time horizon findings.