Having scaled a site to millions of users and billions of data items I think the strategy of extinguishing fires as they arise is the wrong strategy. It will lead to lots of stress, lots of sleepless nights and lots of unneeded bugs. I know, because we used this strategy and I would not recommend it. There's a better way that's less stressful and better for your health.
It's the idea of being proactive and reactive - - and as a good developer you should master both and be able to switch between these states.
For example, backups and anticipations of future load should be proactive. If your load is close to maxing out today then don't wait till your web site is down before you begin optimizations. Know everything about your systems so you can take an educated guess of how many more users you can handle. Think about which strategies you can apply if you want to scale 10x, 100x, 1000x of your current load.
Reactive programming is great when you are exploring new territory and are unsure how many would use something. This is the state where you don't think about scaling to millions of users. This said, as a good developer always keep in mind how something can be scaled or how fast something is.
In other words: it's about finding balance between the proactive state and the reactive state. Don't optimize prematurely, but don't optimize when everything is burning down either. You have lots of data so use this to your advantage and make educated guess about your way of scaling.
Totally agree with what you are saying here, particularly once the idea is validated and you start to see good adoption. I've used todoist and wedoist for a long time and it certainly is reliable ;-)
This post mainly covers the really early stages before millions of users, where the most important thing is focusing on the product and market fit as much as possible. We certainly treated scaling issues in a reactive way at this point. I'd like to think we have got a lot better at being proactive as the service has grown and more people depend on us - for example the recent move to AWS was due to hardware constraints in our old setup, it was planned well in advance and executed with minimal downtime.
Agree. If growth is (reasonably) predictable I think dedicated is a great way to go. Surely if you are aiming for an MVP, you don't want to be worrying about managing multiple boxes and adjusting the amount of RAM, CPU and diskspace they have access to. Better to slightly over provision with a cheap dedicated server then expand when you have a feel for the requirements of the project.
Definitely a good point, however Linode don't offer dedi's and it didn't make sense from a time investment point of view to move away from them until this point.
It's the idea of being proactive and reactive - - and as a good developer you should master both and be able to switch between these states.
For example, backups and anticipations of future load should be proactive. If your load is close to maxing out today then don't wait till your web site is down before you begin optimizations. Know everything about your systems so you can take an educated guess of how many more users you can handle. Think about which strategies you can apply if you want to scale 10x, 100x, 1000x of your current load.
Reactive programming is great when you are exploring new territory and are unsure how many would use something. This is the state where you don't think about scaling to millions of users. This said, as a good developer always keep in mind how something can be scaled or how fast something is.
In other words: it's about finding balance between the proactive state and the reactive state. Don't optimize prematurely, but don't optimize when everything is burning down either. You have lots of data so use this to your advantage and make educated guess about your way of scaling.