ClojureCheck is back!

ClojureCheck is back. It brings specification based testing to Clojure and integrates (almost) seamless with clojure.test. So what does specification based testing mean and how does it help you?

Nostalgia

ClojureCheck is one of my oldest Clojure projects. It started as ClojureTAP -- a testing library producing TAP output. Later on I started to port QuickCheck to Clojure. And hence renamed it. However, I never got really far and things stalled.

I feel very sad about this. It's the best documented Clojure project I ever did and it brings up memories from times where todays require system everyone complains about didn't even exist. So for quite some time now, I had the urge to return ClojureCheck. I just didn't want to let it rot further.

The Revival

Before I started with the work I had to decide where to hook into. Originally ClojureCheck was developed more or less in parallel to clojure.contrib.test-is (now: clojure.test). So it basically contains a similar machinery. However, clojure.test is now the de-facto standard being included in Clojure itself. So there is no reason to do things twice and it hooking into clojure.test is a no-brainer. (Very sad: a lot of nice work gets nuked.)

I also did some more research. I based my original approach too much on QuickCheck. But this gives raise to some problems. In particular Haskell has one more dimension of information: the type system. So you can have one function – arbitrary – which does the right thing depending on the return type you declare. In Clojure this type of dispatch does not really fly.

Tom Moertel's LectroTest is written in Perl which is similar to Clojure in terms of dynamism. I took LectroTest as a source of inspiration for the port to Clojure.

So what is it then?

So after this rather long introduction: What is ClojureCheck?

ClojureCheck provides facilities to easily generate random test data, which are then used to „prove“ certain properties of the code under test. One can eg. generate test data based on a specification. Tom Moertel does this for email addresses in his talk on LectroTest.

The Test Case

Let's use another example from LectroTest: angular-diff.

(ns our.angular.diff
  (:use clojure.test)
  (:require [clojurecheck.core :as cc]))

(defn angular-diff
  [a b]
  (-> (- a b) Math/abs (mod 180)))

angular-diff returns the smallest angle between two given angles a and b. So let's write some tests.

(deftest angular-diff-test
  ; Identical angles should be zero
  (are [a b r] (= (angular-diff a b) r)
     0   0  0
    90  90  0)
  ; Order of angles shouldn't matter
  (are [a b r] (= (angular-diff a b) r)
     0  45 45
    45   0 45
  (is (= (angular-diff 0 270) 90)
      "Should return the smallest angle")
  (let [a (* 360 2)
        b (* 360 4)]
    (is (= (angular-diff a (+ b 23)) 23)
        "multiples of 360 degrees shouldn't matter"))))

And fire:

our.angular.diff=> (run-tests)

Testing our.angular.diff

Ran 1 tests containing 4 assertions.
0 failures, 0 errors.
{:type :summary, :test 1, :pass 4, :fail 0, :error 0}

Yeah! The world is good!

The Evil Counterexample

Is it? Let's apply ClojureCheck to our test problem. We want to generate the problems. So as Tom Moertel says in his talk: we need needles and an easy way to spot them.

We guess some arbitrary angle a. Now just taking a second angle b wouldn't help as very much. So we have somehow to work our way backwards: from the solution to the input.

So next we guess our result: diff. When we feed angular-diff with a and (+ a diff) the expected result is diff. However adding multiples of 360° don't change things. So we guess a winding count n and also add it. So whenever `(angular-diff a (+ a (* n 360) diff))≠(Math/abs diff)` we found our needle!

So let's see how we can cast this in code.

(deftest angular-diff-property
  (cc/property "angular diff returns the smallest angle between a and b"
    [a    (cc/int)
     n    (cc/int)
     diff (cc/int :lower -180 :upper 180)]
    (let [b (+ a (* n 360) diff)]
      (is (= (angular-diff a b) (Math/abs diff))))))

And now let's run our test:

our.angular.diff=> (run-tests)

Testing our.angular.diff

FAIL in (angular-diff-property) (core.clj:305)
falsified 'angular diff returns the smallest angle between a and b' in 3 attempts
inputs where:
  a = 1
  n = -1
  diff = 1
failed assertions where:
  expected: (= (angular-diff a b) (Math/abs diff))
    actual: (not (= 179 1))

Ran 2 tests containing 5 assertions.
1 failures, 0 errors.
{:type :summary, :test 2, :pass 4, :fail 1, :error 0}

Dang! :( The world is bad.

The Fix

But we shouldn't think like this. The world is not bad! We are just too stupid. We should be happy about the failing test, because it tells us, that we did a mistake in our code. And a mistake is always an occasion to learn. (Oh, dear. The QA guy shines through…)

So to fix our function we have to adapt for angles where the difference is greater than 180°.

(defn angular-diff
  [a b]
  (let [diff (-> (- a b) Math/abs (mod 360))]
    (if (> diff 180) (- 360 diff) diff)))

Now a test run should look like this:

our.angular.diff=> (run-tests)

Testing our.angular.diff

Ran 2 tests containing 5 assertions.
0 failures, 0 errors.
{:type :summary, :test 2, :pass 5, :fail 0, :error 0}

Upshot

Testing systems with random input can help. It depends on how you do it. In particular stimulating the system with random input of a pre-defined structure can help finding edge cases hidden deep in the logic.

For unit testing it can help to find blind spots in the input coverage. But with carefully designed tests there might not be any improvements at all.

So in the end: YMMV.

Published by Meikel Brandmeyer on .