People have been suggesting this ever since Copilot was announced and it doesn't work on any level. They're using all code on GitHub, even the ones with no license and which you can't use for any purpose and the reasoning is that they see it as fair use - which supersedes any licenses and copyrights in the US.
They only claimed that training the model was fair use. What about its output? I argue that its output is still affected by the copyright of its inputs, the same way the output of a compiler is affected by the copyright of its inputs.
