I built a fault-tolerant system for binding lots of disparate and network-unaware services via a monitored event-driven communication protocol to a network-fronted cloud architecture. It's called Katamari, and it's the version of Fuzed (this same concept just for Rails instances) that I modified to run as the majordomo of Powerset's realtime query analysis layer. It let us take un-threaded and network-unaware software that talks over local file descriptors and turn them into a cloud service.
Katamari also has some fairly good fault tolerance characteristics. It aggressively tries to maintain a cloud and it has no real single points of failure. It recognizes hung or crashed instances of individual workers and restarts them. It also supports arbitrary versioned resources existing in parallel, which is great when you want to do A/B testing.
It was also designed so that it was usable in both a development and production environment. While currently little slow for most production uses (~2ms overhead for most queries, with a special case of ~12ms), it's plenty fast and runs in less than 10mb of memory on most linux systems, and can transparently go from a 1-box to a N-box solution.
Microsoft open-sourced it for me (after 2 years of bureaucratic nonsense), so I am just pulling out the useless and crufty parts before I put it up on GitHub. I've given 2 talks about the technology, one for Fuzed at Railsconf and one on Katamari at the last Erlang Factory. I hope to have a modernized version maintained this year, and start moving the tech forward.
Katamari is about 10k lines of code; with about 2/3 of that being Erlang and the remaining 1/3 being Ruby code for the glue (Ruby was Powerset's primary glue and binding language, so it is the only supported binding). Katamari aggressively leverages the characteristics of Erlang to accomplish this feature set; its safe to say all the fault tolerance wouldn't have been doable without Erlang's implementation providing the basics for it.
Katamari also has some fairly good fault tolerance characteristics. It aggressively tries to maintain a cloud and it has no real single points of failure. It recognizes hung or crashed instances of individual workers and restarts them. It also supports arbitrary versioned resources existing in parallel, which is great when you want to do A/B testing.
It was also designed so that it was usable in both a development and production environment. While currently little slow for most production uses (~2ms overhead for most queries, with a special case of ~12ms), it's plenty fast and runs in less than 10mb of memory on most linux systems, and can transparently go from a 1-box to a N-box solution.
Microsoft open-sourced it for me (after 2 years of bureaucratic nonsense), so I am just pulling out the useless and crufty parts before I put it up on GitHub. I've given 2 talks about the technology, one for Fuzed at Railsconf and one on Katamari at the last Erlang Factory. I hope to have a modernized version maintained this year, and start moving the tech forward.
Katamari is about 10k lines of code; with about 2/3 of that being Erlang and the remaining 1/3 being Ruby code for the glue (Ruby was Powerset's primary glue and binding language, so it is the only supported binding). Katamari aggressively leverages the characteristics of Erlang to accomplish this feature set; its safe to say all the fault tolerance wouldn't have been doable without Erlang's implementation providing the basics for it.