We are Changing our Developer Productivity Experiment Design
Summary
METR reports on changes to its developer productivity experiments, highlighting challenges in measuring AI-assisted productivity due to selection effects, participant bias, and pay-rate changes. The article revisits prior findings that AI can slow tasks and discusses how ongoing studies may understate true productivity benefits, outlining planned design changes and alternative measurement approaches.