Do code size, instruction scheduling and register allocation matter to you?

Do you have an alternative to LLVM that only provides those four architectures? If not, you can constrain the scope of your LLVM build using -DLLVM_TARGETS_TO_BUILD.