When testing does not cut it

Testing works great, especially when it drives the work of transforming specs into code. But, testing is just about functionality. And functionality is just part of the story that makes a software system.

Let's take an example. Suppose you are writing a system that has a server side based on JBoss. And suppose you want your system to rely on the clustering possibility of JBoss. How would you test that the system works in a cluster as expected? For a complete test, you would need to do something like this:

  • setup a cluster with at least two instances of JBoss running,
  • create multiple clients and lookup all the session beans that you want tested from the cluster,
  • make sure you get a connection to the first node,
  • take that one node down, and
  • check to see that the requests to the already acquired session beans is not failing.

This test would work nicely for the current set of beans. When a new one is created we would have to extend the test with that particular session bean. However, even if we would develop test-first, we might could easily forget to test the clustering requirement simply because it is not a functional one: from a purely functional point of view, a use case could be satisfied even without the clustering in place.

It is not impossible to setup a test for our problem, but it is expensive from multiple perspectives: it is difficult to setup, it is not straightforward to extend it, it takes long to run the test. But, the worst part is that when the test fails, it will be unclear what exactly went wrong.

What is the alternative? First, let's see what does it take to specify clustering in JBoss. It turns out that all you need to do is to annotate the interesting session bean classes with @Clustered. If you accept that JBoss does work as expected, the only thing you need to test is that you have the right annotations in place. This can be checked structurally. With Moose, the check would look like:

model allClasses select: [ :each | 
   (each isAnnotatedWith: 'Stateless') and: [
      (each isAnnotatedWith: 'Clustered') not ]]

Using this rule on a system results in a list with all (stateless in this case) session beans that do not have the required annotation. As long as the list is empty, you are fine. If the result is not empty, you know you have a problem.

The direct cost of building this rule is significantly lower than the one associated with the functional test. But, the best thing is that when the structural rule fails, it provides an immediate pointer to what needs to be fixed.

Testing works great, but only for problems that are best described functionally. For the other problems, you are better served by different analysis tools.

Posted by Tudor Girba at 7 May 2012, 9:30 pm with tags assessment, story, spike, moose link