"where numbers are boxed doubles on the heap, etc."

Actually, you don't know that. Since doubles are immutable, the runtime can (and does) cheat, including turning many of them into integers. It all boils down to predictable usage patterns.