Noah Sussman wrote a post on Things you should test, “A checklist of things that are worth testing in pretty much any software system.”

Many of the things on the list are helpful reminders. However, I think the mindset it encourages is essentially wrong.

The mindset is basically this:

Identify common mistakes that developers make, and ensure you are writing tests that check you haven't made them.

The problem with this approach is that it is essentially whack-a-mole debugging. There is a never ending supply of bugs to kill.

A much more helpful approach is found in this post on easy programming that advocates “Never fix a bug twice” (about one-third of the way down).

If you come across a bug or class of bugs that often occur, you should not be thinking first of all “better add that to my list of classes of bugs that need testing against”. You should rather be thinking “how can I change the system so that this class of bugs disappears entirely?”.

So, to take some of items listed for testing:

• Input handling bugs, such SQL injection and XSS attacks.

  In Django apps, I never write tests for SQL injection attacks or XSS attacks. The reason for this is that these types of bugs are very rare. The reason for that is that the APIs that are easiest to use for executing SQL or generating HTML are secure by default. If there are any errors of this type in my programs, they will be very rare and obscure, and trying to catch bugs by a black box scatter-gun approach will be a waste of time.

  With XSS, there are times when I am slightly more tempted to generate HTML in ways that might be vulnerable to XSS – for example, using Python string interpolation. At this point, however, you still shouldn't reach for a test to ensure you did it correctly. Rather: stop writing HTML generation the stupid way!

  To do that, you first have to ask “why?”. “Why am I tempted to write HTML this way?”. The answer is usually “there isn't a convenient API for the particular way I want to write this”. The correct solution now becomes obvious: create the API that removes the temptation to introduce potentially buggy code.

  So, here is some code in a project I'm writing, that uses string interpolation to implement a template tag for formatting a link in a standard way:

  from django.core.urlresolvers import reverse
  from django.template import Library
  from django.utils.html import escape
  from django.utils.safestring import mark_safe
  
  register = Library()
  
  @register.filter
  def account_link(account):
      return mark_safe(u'<a href="%s" title="%s %s">%s</a>' % (
              escape(reverse('account_stats', args=(account.username,))),
              escape(account.first_name),
              escape(account.last_name),
              escape(account.username),
              ))
  

  The problem with this is that I have to remember to use escape on each variable. The reason I wrote it this way is that Django's Template API is kind of bulky for this use case. So, I should instead have written this:

  from django.core.urlresolvers import reverse
  from django.template import Library
  
  from somewhere import html_fragment
  
  register = Library()
  
  @register.filter
  def account_link(account):
      return html_fragment(u'<a href="%s" title="%s %s">%s</a>',
              reverse('account_stats', args=(account.username,)),
              account.first_name,
              account.last_name,
              account.username,
              )
  

  And then write the API, html_fragment, that makes this work, which is just:

  from django.utils.html import escape
  from django.utils.safestring import mark_safe, conditional_escape
  
  def html_fragment(template, *args):
      return mark_safe(template % tuple(map(conditional_escape, args)))
  

  EDIT: After some discussion on django-devs and subsequent modification, this is now in Django 1.5 as 'django.utils.html.format_html', so use that instead of the above

  I'm now no longer tempted to write it the vulnerable way, and I don't need a test (though I may want a test or two for html_fragment). I have two ways of doing it - html_fragment for very small snippets, and Django templates for bigger chunks, both of them secure by default.

  So, if you find yourself needing tests for specific SQL injection or XSS attacks in your code, you are probably doing it wrong. Fix the underlying API that makes the mistake likely in the first place.

• Input handling type checking - “Invalid values such as null and NaN. Strings instead of integers, arrays instead of strings.”

  Input handling should generally only occur on trust boundaries, but it should occur systematically on such boundaries. If you have a problem with the wrong type of value being passed to an internal function, due to an external input being passed in, you shouldn't be dealing with it in the function, you should be asking “how was this even possible?”.

  One solution of course is a static type system. Of course, that might not be possible/practical in your situation, but you should be thinking about it.

• Math-related bugs: in many of the listed instances in Noah's post, the underlying bug is that the language does funny things.

  So you should be asking “why am I using this language? Is it the right tool for the job?”.

  Or at least “will a library solve this problem?”.

  If you are handling decimal values, the solution is a library that does it well, and doesn't make it easy to make mistakes. For example, the Python 'decimal' library does not allow multiplication of floats and decimals - which sounds inconvenient, but stops you from making all kinds of mistakes.

• Units of measurement.

  If you have potential bugs with this, you should be thinking “why are these possible? How can I eliminate the bug entirely?”.

  Different languages have different solutions, often involving including the unit of measurement in the value itself. Haskell has various solutions that make it impossible to add "3 years" to "2 meters", for example, and the compiler will stop you at compile time.

  For Python, there are things like magnitude and units that work at run time.

  Of course, even if you use these internally, you'll still have boundaries where you need to do some conversions from inputs etc., and at this point you might need to know what units the inputs are in. But:

  1. You should try to insist that the external system sends the unit information along with the numerical value, so that you never have to rely on common knowledge, and you will know immediately if something changed. And your internal APIs have helped you do this by forcing the issue.

  2. Even if the external systems can't be changed, you'll still have very few places to check - just your input and output boundaries, each of which should be managed in a single place. And writing tests probably won't help you very much – you should write at most one, as a single integration test. Your time will be better spent manually checking the boundaries.

• Text - “Are Unicode inputs treated differently than ASCII?”

  Again, you should be using proper internal datatypes – unicode everywhere – to eliminate potential bugs.

  Python 2.x is notorious here - its ‘forgiving’ conversion between bytestrings and unicode causes endless bugs that are found in production, not in development.

  And the correct solution is not just to write tests – which you may have to do for now – but rather to fix the underlying cause. This is what has been done in Python 3.

The same type of things applies to almost everything on the list, and in the cases where I can't see how it applies, I imagine that this is due to my own lack of imagination :-) If I can't think of a way of completely eliminating a class of bug, then I should think harder, rather than assume it is impossible.

Notice that from the perspective of an older programmer, there are some notable omissions from the list - things like buffer overflows, for example. Presumably that's because the author uses languages/frameworks where those bugs are very unlikely. And that has happened not because previous generations of programmers wrote lots of tests, but because they wrote languages and systems where those bugs are impossible or almost impossible.

So, in all this I'm not saying that you won't ever have to write tests for these kinds of bugs. But every time you identify a class of bugs, tests are a very poor answer. You should try to ensure that you never fix the same bug twice, and so never write the same test twice. If you are writing essentially the same test more than once, you are proving that you haven't fixed the underlying issue yet.

If a bug is worth adding to a list of common bugs, then you have identified a systemic problem with your platform, and it is worth eliminating entirely. We should be striving for libraries/languages/systems which do that.