We are Changing our Developer Productivity Experiment Design
Summary
METR reports on changes to their developer productivity experiment design, highlighting mixed evidence of AI impact on productivity due to selection effects and task design. The study finds that prior results showing slowdowns may be biased, and newer data suggest higher potential uplift but remain inconclusive due to participation biases and wage-rate changes. The article outlines methodological challenges and multiple avenues for improved measurement, including fixed-task experiments and observational data.