This article misses the elephant in the room: how to define "reliability". Most software perceived to be reliable (eg, MS Paint, bash) have numerous bugs, it's just that no one comes across them. Software that is perceived to be "reliable enough" (eg, MS Word, Chrome), have bugs that pop up regularly.
Here are some vague uptime reliability measures and what you need to get there:
- 70% reliability: you can often just free-code it and pray.
- 95% reliability: you better have a bunch of tests.
- 99.5%: you need automated testing, coverage tools, documentation, team coordination tools, and and solid merge/deployment process.
- 99.99%: all-of-the-above plus tests that aggressively attack and analyze production systems (eg, Netflix Chaos Monkey), coverage tests that do branch coverage and not just line coverage, simulating network failures, and/or test on a variety of hardware and user-configurations.
- 100%: The only way to be absolutely certain, is to convert each line of code into a mathematical operator and solve for a formal proof of correctness.
The 100% reliability standard is not just theoretical. The U.S. government issues software standards for safety critical devices (military, aviation, healthcare), and some of them do indeed require mathematical proof of correctness (EAL6+ being a key standard). They generally do not require 100% bug-free code, but just that some particular features are 100% bug free (e.g., the OS/scheduler will never deadlock).
> and some of them do indeed require mathematical proof of correctness
> (EAL6+ being a key standard)
Nitpick: EAL6 is still semi-formal, EAL7 is formally verified design and tested. EAL7 is very rare, on the CC certified products page [1] there are only 2 EAL7 certified products: a hardware diode, and a virtual machine OS.
> Nitpick: EAL6 is still semi-formal, EAL7 is formally verified design and tested. EAL7 is very rare, on the CC certified products page [1] there are only 2 EAL7 certified products: a hardware diode, and a virtual machine OS.
You are definitely correct. Also, disclosure... I worked on that EAL7 Virtual OS. When I left it was only EAL6+, likely leading to my bias towards it.
Here are some vague uptime reliability measures and what you need to get there:
- 70% reliability: you can often just free-code it and pray.
- 95% reliability: you better have a bunch of tests.
- 99.5%: you need automated testing, coverage tools, documentation, team coordination tools, and and solid merge/deployment process.
- 99.99%: all-of-the-above plus tests that aggressively attack and analyze production systems (eg, Netflix Chaos Monkey), coverage tests that do branch coverage and not just line coverage, simulating network failures, and/or test on a variety of hardware and user-configurations.
- 100%: The only way to be absolutely certain, is to convert each line of code into a mathematical operator and solve for a formal proof of correctness.
The 100% reliability standard is not just theoretical. The U.S. government issues software standards for safety critical devices (military, aviation, healthcare), and some of them do indeed require mathematical proof of correctness (EAL6+ being a key standard). They generally do not require 100% bug-free code, but just that some particular features are 100% bug free (e.g., the OS/scheduler will never deadlock).