I don't know what to do with tests, to be honest. I am a very inexperienced developer, and the general sentiment is "knowing what to test is a learned skill"
damn.
I could use some opinions actually. The current workplace arguments about what we should or shouldn't test go like this:
"We write tests to establish specifications, then we write code to meet
those specifications. We should have tests covering every aspect of the
system as a way to assert that things are meeting the designed specs."
"but our model objects have allot of tedious detail. Why should we write
tests about what the name field can and can't accept? It's a max length of
32, that's it. Asserting that the constraint fails is beyond tedious and a
waste of our time. Most of the code we'd be testing is auto-generated. Are
we running tests on our framework or our code?"
"Some of this does seem like a waste of time, but what about when
someone changes something as part of the development cycle? eg.
Someone needs to remove validation from the name field to test other
parts of the systems behavior with an overly large name. If we had a
complete testing system in place it would alert us that the model isn't
meeting the spec. We've already had instance where temporary code to
bypass authorization was accidentally pushed to the develop branch."
On one end of the spectrum there is this base level of tests that are actually REALLY useful when developing features. Doing simple testing has helped me catch off by one errors and other kinds of bugs, I can stand by my code confidently. These kinds of ad-hoc in development tests don't cover everything, but they're useful. They're like a substitute for step through debugging to verify code.
On the other, I don't know how big of an issue bug regressions will be, we've heard other developers suffering for lack of a solid testing base to detect problems. Without a full battery of tests asserting everything we know about the spec, there's no way we will catch many regressions.
> I don't know what to do with tests, to be honest. ... HN what do you think?
Well, since you asked!
The most valuable tests in any system is functional tests. Unfortunately most tests are unit tests that give little value.
Here's something that I encountered recently in the wild. Imagine a situation like this, a calculation process with three distint black boxes of code.
Block A - Pull data from 3rd party source
Block B - Distribute data to individual nodes (multi-threaded environment)
Block C - Individual nodes processing data
Each one of these blocks is around fifty thousand lines of code, each one of these blocks has hundreds if not thousands of unit tests. Every line of code is tested by a unit test. Yet there is a very real and dangerous bug in the system.
Why? Because the object passed from B-C has a transient variable and when it is serialized and deserialized that value is lost and reverted to the default value. Now most places leave this value as the default, so it's only one out of a thousand customers that have this problem, but when they do it's a huge problem.
Functional tests is the process of saying "I put meat/spices on this side and sausage comes out the other" it doesn't try to determine the RPM of the grinder, the number of bolts, the rotation of the gears, or the type of metal used to build the sausage-maker. It is simply the process of determining that meat becomes sausage. Functional tests have high value and are absolutely worth knowing.
In most CRUD applications Unit Tests generally wind up testing that the universe still exists as expected (true is still equal to true, zero is not one, and a null value is not the same a string, oh happy days the system works!).
That's not always the case. If you are working on something that is more advanced, say something that should measure the chance a hurricane might knock down a given 3d model you'll wind up having a huge stack of unit tests, and these will be very valuable. More often than not you'll know they are valuable as they'll be spent proving the science behind the application, and not simply that the application 'does something'.
That bug with the transient variable. You could only would really catch that if you had e2e or integration tests covering a majority of the code in your application, right? Even then, only if you were to persist the data, read the data back out, then again run more tests on it.
I accept testing isn't a silver bullet, but ouch.
As a general strategy, it sounds like it's better to run your higher level tests as a gauntlet (feeding the result of test one into test two) then with tightly controlled inputs (using explicit data for each test).
FWIW, here are a few things about automated testing that I’ve found to be true very generally.
Other things being equal, testing is good and automated testing is more valuable than manual testing, for the same reasons that having any systematic, repeatable process is generally more efficient and reliable than doing the same things manually and hoping to avoid human error. Many of the following notes are just reasons why other things aren’t always equal and sometimes you might prefer other priorities.
Automated test suites get diminishing returns beyond a certain point. Even having a few tests to make sure things aren’t completely broken when you make a change can be a great time-saver. On the other hand, writing lots of detailed tests for lots of edge cases takes a lot of time and only helps if you break those specific cases. For most projects, a middle ground will be better than an extreme. But remember that as a practical reality, an absurdly high proportion of bugs are introduced in those quick one-line changes in simple functions that couldn’t possibly go wrong, so sometimes testing even simple things in key parts of the code can be worthwhile.
Automated test suites have an opportunity cost. Time you spend writing and maintaining tests is time you’re not spending performing code reviews, or formally proving that a key algorithm does what you think it does, or conducting usability tests, or taking another pass over a requirements spec to make sure it’s self-consistent and really does describe what you want to build. These things can all help to develop better software, too.
Automated test suites do not have to be automated unit test suites. For example, higher-level functional or integration tests can be very useful, and for some projects may offer better value than trying to maintain a comprehensive unit test suite.
Unit tests tend to work best with pure code that has no side effects. As soon as you have any kind of external dependency, and you start talking about faking a database or file access or network stack or whatever other form of I/O, unit testing tends to become messy and much more expensive, and often you’re not even testing the same set-up that will run for real any more.
A corollary to the above is that separating code that deals with external interactions from code that deals with any serious internal logic is often a good idea. Different testing strategies might be best for the different parts of the system. (IME, this kind of separation of concerns is also helpful for many other reasons when you’re designing software, but those are off-topic here.)
Modifying your software design just to support unit tests can have serious consequences and can harm other valuable testing/quality activities. For example, mechanics that you introduce just to support unit testing might make language-level tools for encapsulation and modular design less effective, or might split up related code into different places so code reviews are more difficult or time-consuming.
Automated testing is there to make sure your code is working properly. It is not a substitute for having robust specifications, writing useful documentation, thinking about the design of your system, or, most importantly of all, understanding the problem you’re trying to solve and how you’re trying to solve it.
No-one really only writes code that is necessary to pass their tests. Even if they religiously adhere to writing tests first, at some point they generalise the underlying code because that’s what makes it useful, and the test suite didn’t drive that generalisation or verify that it was correct. In TDD terms, the tests only drive the red/green part, not the refactor part.
For similar reasons, just because someone has a test suite, that does not mean they can safely refactor at will without thinking. This may be the most dangerous illusion in all of TDD advocacy, but unfortunately it seems to be a widespread belief.
A lot of “evidence” cited in support of various test strategies and wider development processes is fundamentally flawed. Read critically, and be sceptical of conclusions that over-generalise.
And finally, what works for someone else’s project might not be a good choice for yours, and something that didn’t fit for someone else might still be useful for you. If you’re experimenting, it can be very informative just to keep even basic records of roughly how much time you’re really spending on different activities and what is really happening with interesting things like speed of adding new features or how many bugs are getting reported in released code. You’re allowed to change your mind and try a different approach if whatever you’re doing right now isn’t paying off.
damn.
I could use some opinions actually. The current workplace arguments about what we should or shouldn't test go like this:
On one end of the spectrum there is this base level of tests that are actually REALLY useful when developing features. Doing simple testing has helped me catch off by one errors and other kinds of bugs, I can stand by my code confidently. These kinds of ad-hoc in development tests don't cover everything, but they're useful. They're like a substitute for step through debugging to verify code.On the other, I don't know how big of an issue bug regressions will be, we've heard other developers suffering for lack of a solid testing base to detect problems. Without a full battery of tests asserting everything we know about the spec, there's no way we will catch many regressions.
HN what do you think?