Try this: we don't get alien visitors because space travel is prohibitively expensive. But communication is cheap. So you'd expect the sky to be filled with messages, maximally compressed and/or encrypted, and so indistinguishable from noise: background radiation.
The only catch here is that such messages would be prohibitively costly (power wise) to transmit uniformly in all directions. They would be point to point to reduce the power, required to transmit, and increase privacy.
It's difficult to intercept line of sight communications and it's unlikely we'd fall on the line of sight of two hot spots of intelligent communication in all the vastness of space.
Even if they broadcast, I doubt it would be so completely uniform in all directions through long periods of time so as to appear as background.
They are point to point. There's just so many, billions more than the number of stars (imagine cell phones radiating point to point), and even alien technology can't prevent beam-spread over billions of light years (another assumption).
I like it, but: I guess you're talking about civilizations deliberately communicating with each other. If they don't already know each other, then maximal compression or encryption would prevent them recognising each other's signals, by your argument. If they do, then how did that happen?
