If loading the rest of the code and initialization time really matters, yes. But then you're back to square one: you now have to create a CLI to access the server and that new CLI must load quickly. Which would also be the case if you just wrote it that way first, then loading the rest of the code once the command is determined.

And if you're in some project similar to Django, you would have first to query the server to know the possible subcommands...

So what you say makes sense when other parts of the code are slow to initialize and must be accessed frequently, not for the CLI itself.