Strictly speaking, Twitter does not check what you "have" - it only checks that you "know" the secret key. If I stole your phone, dumped all info there and then returned the phone to you - I still could use the private key to fool Twitter into thinking I'm you, couldn't I?
The key is just harder to steal because it is big and is not sent out. But this doesn't seem to have much to do with phones...
You've just described a physical token duplication attack. A consumer phone certainly is easier to attack than a SecurID or smartcard, but it's a far sight from a really really long password. For starters, the challenge response is calculated by the phone's hardware, so that the private key is not exposed.
The "what you know"-type authentication is literally what you know, not "I don't know it but it's written down on my phone, hang on a sec". You're supposed to be able to provide it without reference to notes (or Post-Its stuck to the bottom of keyboards).
The key is just harder to steal because it is big and is not sent out. But this doesn't seem to have much to do with phones...