Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase
Summary
Databricks shares their internal benchmark evaluating coding agents on a multi-language codebase. Key findings show a Pareto frontier across OpenAI, Anthropic, and open-source models, GLM 5.2 performing well, and that price-per-token does not reliably predict end-to-end task costs. Harness choice significantly affects efficiency and cost, with practical implications for selecting models and harnesses in real-world coding work.