It's not that much, actually. Some of us have processed 50-70 million transactions per day on a much smaller infrastructure (25..40 AWS boxes) - quite often designed and deployed just days prior to the massive traffic spike, with fancy features like real-time fraud checking etc.
It's not that hard to do what Twitter does. At least, with the amount of financial backing that they have, I don't expect them to be failing in a spactacular fashion periodically.
The reason for the mismatch is that it's not 70 million transactions a day. It's potentially 70 million times the average number of people that access each tweet, which is probably more like 100+, plus all the polling that twitter applications do even when there are no new tweets, as well as all the ancillary stuff that's required to run something like this (spam filtering, API servers, etc).
Yes. Yes, on average, a given tweet reaches ONE HUNDRED PEOPLE. Do any of you folks even listen to yourself? Even if you change that to "users" instead of "people," since so many twitter accounts are robots - even then, 100 users on average for every tweet is ABSURD.
> the polling that twitter applications do even when there are no new tweets
Polling with no new tweets is far easier to optimize than polling when there are tweets. So just discount this almost entirely in your roughshod analysis.
> spam filtering
Have you been on twitter? We all get spam and see fake accounts trying to get at us regularly. Try mentioning you're on a diet on twitter and see how many followers you rack up. If they're filtering spam, they're not filtering it well, so I hope they aren't burning too much cash on it.
> API servers
API servers are frontends that use dramatically fewer resources than the work the backends have to do. If Twitter's frontends are costing them much money, again, they're wasting money.
I don't see what's hilariously off about that? If you look at the average number of people that a given real person follows, 100 probably isn't that far off. I'm not saying those are actually read by a human, but associated with that many, sure.
My point was that the grandparent was probably an order of magnitude or more off counting 70 million new messages per day as 70 million transactions per day.
I'm not saying they're doing things well, but it's pretty naive to think that you could build Twitter's infrastructure with <50 AWS instances.
You think the API is a trivial matter of spinning up some front end machines? From that presentation linked to above, they were getting 6 Billion API calls per day, or 70k/sec.
Besides that, all of these numbers are 2 years old, and according to those slides, they were growing at about 10x per year. It has probably slowed, but those numbers may all be much bigger now.
I don't know how much it takes Twitter (even though it should be relatively easy to find out), but I do have an idea how much it should take.
Twitter presentations suggest 70 million messages per day. 800 tweets per second:
http://www.slideshare.net/raffikrikorian/twitter-by-the-numb...
It's not that much, actually. Some of us have processed 50-70 million transactions per day on a much smaller infrastructure (25..40 AWS boxes) - quite often designed and deployed just days prior to the massive traffic spike, with fancy features like real-time fraud checking etc.
It's not that hard to do what Twitter does. At least, with the amount of financial backing that they have, I don't expect them to be failing in a spactacular fashion periodically.