The lack of CUDA support on AMD is absolutely not that AMD "couldn't" (although I certainly won't deny that their software has generally been lacking), it's clearly a strategic decision.
Supporting CUDA on AMD would only build a bigger moat for NVidia; there's no reason to cede the entire GPU programming environment to a competitor and indeed, this was a good gamble; as time goes on CUDA has become less and less essential or relevant.
Also, if you want a practical path towards drop-in replacing CUDA, you want ZLUDA; this project is interesting and kind of cool but the limitation to a C subset and no replacement libraries (BLAS, DNN, etc.) makes it not particularly useful in comparison.
They've already ceded the entire GPU programming environment to their competitor. CUDA is as relevant as it always has been.
The primary competitors are Google's TPU which are programmed using JAX and Cerebras which has an unrivaled hardware advantage.
If you insist on an hobbyist accessible underdog, you'd go with Tenstorrent, not AMD. AMD is only interesting if you've already been buying blackwells by the pallet and you're okay with building your own inference engine in-house for a handful of models.
Even disregarding CUDA, NVidia has had like 80% of the gaming market for years without any signs of this budging any time soon.
When it comes to GPUs, AMD just has the vibe of a company that basically shrugged and gave up. It's a shame because some competition would be amazing in this environment.
Nvidia has a sprawling APU family in the Tegra series of ARM APUs, that span machines from the original Jetson boards and the Nintendo Switch all the way to the GB10 that powers the DGX Spark and the robotics-targeted Thor.
The CPUs in their SOCs were not up to snuff for a non-portable game console until very recently. They used (and largely still do I believe) off the shelf ARM Cortex designs. The SOC fabric is their own, but the cores are standard.
In performance even the aging Zen2 would demolish the best Tegra you could get at the time.
You should note that the Switch, the only major handheld console for the last 10 years, is the only one using a Tegra.
And from everything I've heard Nvidia is a garbage hardware partner who you absolutely don't want to base your entire business on because they will screw you. The consoles all use custom AMD SOCs, if you're going to that deep level of partnering you'd want a partner who isn't out to stab you.
There has been a rumor that some OEMs will releasing gaming oriented laptops with Nvidia N1X Arm CPU + some form of 5070-5080 ballpark GPU, obviously not on x86 windows so it would be pushing the latest compatibility layer.
PlayStation and Xbox are two extremely low-margin, high volume customers. Winning their bid means shipping the most units of the cheapest hardware, which AMD is very good at.
Agreed on ZLUDA being the practical choice. This project is more impressive as a "build a GPU compiler from scratch" exercise than as something you'd actually use for ML workloads. The custom instruction encoding without LLVM is genuinely cool though, even if the C subset limitation makes it a non-starter for most real CUDA codebases.
ZLUDA doesn't have full coverage though and that means only a subset of cuda codebases can be ported successfully - they've focused on 80/20 coverage for core math.
Supporting CUDA on AMD would only build a bigger moat for NVidia; there's no reason to cede the entire GPU programming environment to a competitor and indeed, this was a good gamble; as time goes on CUDA has become less and less essential or relevant.
Also, if you want a practical path towards drop-in replacing CUDA, you want ZLUDA; this project is interesting and kind of cool but the limitation to a C subset and no replacement libraries (BLAS, DNN, etc.) makes it not particularly useful in comparison.