A while ago I was invited by the lynch-mob in #twisted to write a post about what I meant when I said that "Unit tests are not real tests", and here it is.
Now, recently there has been much said about Unit-testing, and code coverage, and even sensationalist blog post titles, which in retrospect "Unit tests are not real tests" would have been. I won't talk about code test coverage here, because its not relevant, we should strive to get as much coverage as we can. And of course I believe that unit tests are a fundamentally important piece of development. Especially since I tend to use Python, which is prone to disaster if not tested correctly.
So, if unit tests are real tests, then why did I say they aren't? I deal mostly with application development. The user of the application is a human being. Now quite simply, the only true test of my application is the user interacting with it. I understand now that this is not really the case for library development, where the user is a piece of code, and I guess you can have true tests for it that are unit tests.
So how do we do it? Well, I am not sure what kind of testing this is, but I call it "Acceptance Testing". It is evidence that the application behaves as advertised. I guess it would come under headings like "Verification". We have a list of written tests that go something like:
|Test||Expected Outcome||Actual Outcome|
|User double-clicks the icon||log in screen displayed|
|User enters date in the future||text entry goes red and user is informed|
And they go on and on, we have hundreds of pages of these. As I was in a start-up consisting of one person, I used to do these myself. Now the test department does it.
But these tests are just another type of unit test! Yes they are, and probably purists would disagree, but look at them:
- You test every single unit of an application (but from the user perspective)
- When you find a bug, you write a new test to test for it, then fix it
- When you add a new feature, you write the tests first (TDD!)
- Anyone can run the tests
Can't you automate these? Yes you can, but that would again slip them into the realms of "non-real tests". As I said, a human user has to test the application in order to reflect the actual user of the software. I really love automation. I have hacked on kiwi's automated test-runner, played with various other tools. Again, like traditional code-based unit tests, they are great tools.
So, I should really amend my statement to the #twisted guys to something more like:
"Code-based unit tests are not the true tests for an application that interacts with a human user."