Why test at all?

It would be great if we were all infallible, wouldn’t it? Imagine if everyone always did everything perfectly ‘right’ on their first attempt. Actually, come to think about it, that would be terrible! A life with no challenge – we could all drive like Lewis Hamilton on our first try, paint like Da Vinci, flip beer mats like my mate Dave. Life would be dull dull dull.

Luckily, that’s not reality – Lewis Hamilton can’t paint beer mats like Da Vinci (note – I have no proof of this either way but I think it’s a fair assumption) – so life is full of challenge and interest and human fallibility and we should embrace that. But it does mean that there are times where we need to be careful, check things are correct, ‘measure twice cut once’ etc.

Creating software is one of those times.

A mistake or a misunderstanding can happen at pretty much any point in the software development process. Sometimes these are little things that actually don’t have a big impact. Sometimes these are big whopping things that have a massive impact. Sometimes these are tiny little things that have a colossal life-threatening impact.

We obviously don’t want the high impact bugs. These we need to avoid. But what about the low impact ones? Well even these will need fixing eventually, and that means re-work, which means time is wasted.

So, we test to try to avoid mistakes impacting the end users – no matter who they are and how much they are impacted by the mistake.

When to test

Now, we could just test at the ‘end’ of the software development process – catch all those mistakes at the same time. That makes sense, right? Well, there are a few issues with that…

A wise man once said “a mistake or a misunderstanding can happen at pretty much any point in the software development process” (it was me a few paragraphs above). If you make a mistake early, everything you do after that could be affected by that mistake. That could mean a lot of work needs to be changed or even started from scratch, all because an early mistake went un-noticed for a long time. So, the first strike against just doing all the testing at the end: early mistakes going un-noticed could lead to a lot of re-work.

The more distance that you have when testing something from the creation of the thing you are testing, the greater the chance that you will miss something when testing. That distance could be time – ie you test everything at the end. But it could also be communication – ie the person testing was not involved in the creation of the ‘thing’ and so doesn’t understand it very well, thus is more likely to miss an issue. Here we have the second strike: the greater the distance (time or communication) between creation and testing, the greater the chance that a mistake will be missed.

Related to the previous issue, you also need to consider how you know what to test when you are testing everything at the very end. Do you have a big long list of tests to perform? When was that list written? Has anything changed in the meantime? Are we going to assume testers are infallible in a way that developers are not? I think here you are faced with a choice:

•   Stick with your big long list of tests and hope that nothing has changed, inevitably leading to missing bugs, or
•   Spend time re-working your big long list of tests, catching up on everything that has changed and hoping nothing gets missed, using a load of extra time and still missing bugs

Thus, we have our third strike: big long lists of tests performed at the very end waste time and are likely to miss mistakes that have been made.

Three pretty good reasons not to test everything at the very end of the software development process – and there will be more than I have listed. So, what’s the alternative?

Its obvious really – do the opposite! Test as early and as often as possible. Find those early mistakes that you make soon after you make them and avoid the re-work. Test when everything is still fresh in people’s minds so that things don’t get missed. Write lists of tests when you know what you need to test and save time and avoid missing things.

In essence, by bringing the testing earlier rather than leaving it to the very end you are ‘shifting the testing left’. Shift-left testing deserves a blog article of its own, so I’ll leave this section with a simple statement:

Test early and often.

Manual vs automated testing

I think we have already established that humans are fallible. Here’s some more bad news… they are a bit slow too. No, I don’t mean stupid (though some are). I actually mean slow slow.

We have also established that we should be testing early and often.

So testing is something we want to do a lot of, something we want to do frequently throughout the development process and something that we want to be consistently repeatable. If we are running ‘a lot of’ tests slowly and we are doing that ‘frequently throughout the development process’ it stands to reason that the need to test will slow down delivery. Does that matter?

If it takes a long time to test from the point where a developer thinks they have completed their work, then it also takes a long time to get feedback to the developer. Then you need to triage any defects and then fix them. Its all less fresh in the developer’s mind because it’s been a while since they wrote that code so the chance of the developer making a mistake is increased leading to more defects and more looping around.

And don’t forget, the developer has been pulled off what they were doing to fix these defects, so now when they go back to what they were doing they have to context switch again and there is even more opportunity to make mistakes, again leading to more defects and more looping around.

While all of this is going on, the new ‘thing’ is not being delivered to the end users and, let’s not forget, ‘time is money’.

Clearly, ‘slow and prone to mistakes’ are qualities that are somewhat at odds with what we want from testing. If only there was some way to make it faster and consistent!

This is, of course, where test automation comes in. Automated tests run faster and they run the same every time – just the qualities we were looking for. But it takes time to create automated tests and not all automated tests are the same. We need to identify which tests to automate, where to automate them and how. And we need to make sure we invest our time in automating the tests that provide the most value – the ones that provide the best ‘bang for buck’.

The Test Pyramid

Luckily, there is a guide to investing your time in the right types of tests… and its pyramid shaped.

The thinking is this:

•   Automated unit tests are the closest to the code being created – ie they run first. They are the least brittle and are the fastest to run. The majority of your test creation time should be put into unit tests.
•   Automated service tests – API tests, contract tests etc – run next. They take longer to run than unit tests. Many problems should already have been caught by the unit tests when the service tests run. Therefore, less time should be invested in service tests than unit tests.
•   Automated UI tests run last. They are the slowest to run and are the most brittle. Most problems should have already been caught by the unit and service tests. Therefore, UI tests should make up the smallest part of your automated test suite.
•   Manual testing, such as exploratory testing, should still be a part of the tests that you run, but once the automated tests have run very few problems should be left to find. Therefore, you need to invest the least amount of time into manual testing.

Summary

So, what have we covered?

1.   Testing is important
2.   We should test early and often
3.   Manual testing is slower and more prone to errors than automated testing
4.   Automated tests should be weighted towards the tests that run early and quickly
5.   Lewis Hamilton can’t paint beer mats like Da Vinci