All the time machines that seem possible in current physics require massive objects and must exist before the earliest point in time to which you can travel. So, the simplest explanation for why there are no time travelers is that we haven't constructed a time machine for them to use to visit us, yet.
I've heard that argument before, but people much more knowledgeable on the issue than you or I (no offense) always seem to obsess on the time paradox issue, and to a lesser extent the radiation feedback loop that could be created from an expanded wormhole, as the only stopping blocks:
Also, see "massive object".