This is far, far more rigorous than the experiment Microsoft behind Microsoft's claim that Copilot made devs 55% faster.
The experiment in question was to split 95 devs into two groups and see how long it took each group to setup a web server in Javascript. Control took a little under 3 hours on average, the copilot group took 1 hour and 11 minutes on average.
And it is thanks to this weak experiment that Github proudly boasts that Copilot makes devs 55% faster.
By contrast the conclusion that Copilot makes devs ~25% more productive seems reasonable, especially when you read the actual paper and find out that among senior devs the productivity gains are more marginal.
