Tutorial

This tutorial mimics the one for QuickCheck++.

The examples here can be found in the tutorial directory.

Using AutoCheck

AutoCheck was written for C++11 and leverages several new features and standard libraries. Be sure you compile with C++11 enabled (in both the language and the standard library).

#include <autocheck/autocheck.hpp>
namespace ac = autocheck;

Defining Properties

A property is simply a callable object returning a bool from its arguments. It can be a function:

template <typename Container>
bool prop_reverse_f(const Container& xs) {
  Container ys(xs);
  std::reverse(ys.begin(), ys.end());
  std::reverse(ys.begin(), ys.end());
  return xs == ys;
}

... or a function object:

struct prop_reverse_t {
  template <typename Container>
  bool operator() (const Container& xs) const {
    return prop_reverse_f(xs);
  }
};

... or even a closure built from a lambda expression:

auto prop_reverse_l = [] (const auto& xs) {
  return prop_reverse_f(xs);
};

Properties can take arguments by const or non-const reference. Function objects can have const or non-const function-call operators.

Verifying Properties

To verify a property, pass it to autocheck::check. At minimum, a few things must be specified:

• The parameter types of the property.
• The property.

ac::check<std::vector<int>>(prop_reverse_t());

check will randomly generate test cases, starting small and growing larger, until either the desired number of tests pass or one fails.

OK, passed 100 tests.

By default, AutoCheck will run 100 tests. We can change that:

ac::check<std::vector<int>>(prop_reverse_t(), 300);

OK, passed 300 tests.

In case of failure, AutoCheck will display the counterexample:

struct bad_prop_reverse_t {
  template <typename Container>
  bool operator() (const Container& xs) const {
    Container ys(xs);
    std::reverse(ys.begin(), ys.end());
    // Reversed only once!
    return xs == ys;
  }
};

ac::check<std::vector<int>>(bad_prop_reverse_t());

Falsifiable, after 5 tests:
([1, -1])

Preparing Test Cases

Consider a function for inserting an element into a sorted vector:

template <typename T>
void insert_sorted(const T& x, std::vector<T>& xs);

We can specify a property testing that the vector remains sorted after:

struct prop_insert_sorted_t {
  template <typename T>
  bool operator() (const T& x, std::vector<T>& xs) {
    insert_sorted(x, xs);
    return std::is_sorted(xs.begin(), xs.end());
  }
};  

However, verifying this property will fail eventually:

Falsifiable, after 7 tests:
(0, [3, 2, -2])

The problem is that insert_sorted assumes its input is sorted, but the generator gives it random unsorted vectors. We can change the test case generator (a model of the Arbitrary concept) for a set of tests. With this method, we can add a premise that will filter out inputs:

auto arb = ac::make_arbitrary<int, std::vector<int>>();
arb.discard_if(
    [] (int, const std::vector& xs) {
      return !std::is_sorted(xs.begin(), xs.end());
    });

This still won't be quite satisfactory:

Arguments exhausted after 12 tests.

The output means all the tests passed, but we didn't hit our goal (100 tests passed) before hitting a threshold (500 tests discarded, by default). We can increase the threshold to accomodate:

arb.at_most(2000);

Arguments exhausted after 15 tests.

This isn't helping much. What we really need is preprocessing:

arb.prep(
    [] (int, std::vector<int>& xs) {
      std::sort(xs.begin(), xs.end());
    });

OK, passed 100 tests.

Custom Generators

As an alternative to preprocessing, we can define custom generators. They have the benefit of being more reusable, but they require a bit more code.

A Generator produces an infinite sequence of values. The minimal interface is very simple:

concept Generator {
  typedef ... result_type;
  result_type operator() (size_t size);
};

We can easily write a custom generator for sorted vectors:

template <typename T>
struct ordered_list_gen {
  ac::generator<std::vector<T>> source;

  typedef std::vector<T> result_type;

  std::vector<T> operator() (size_t size) {
    result_type xs(source(size));
    std::sort(xs.begin(), xs.end());
    return std::move(xs);
  }
};

ac::check<int, std::vector<int>>(prop_insert_sorted_t(), 100,
    ac::make_arbitrary(ac::generator<int>(), ordered_list_gen<int>()));

OK, passed 100 tests.

AutoCheck actually provides a standard generator for sorted vectors:

ac::check<int, std::vector<int>>(prop_insert_sorted_t(), 100,
    ac::make_arbitrary(ac::generator<int>(), ac::ordered_list<int>()));

AutoCheck also provides a few combinators to make it easier to build custom generators.

Inspecting Test Case Distribution

When dealing with random test cases, we need to make sure we're hitting all the interesting cases. We can classify different test cases and have the distribution printed if they all pass.

Classification is performed by a [[classifier]]:

ac::classifier<int, std::vector<int>> cls;

To begin, we can check whether a test case makes the property trivial. For sorted insertion, that occurs when the vector is empty:

cls.trivial(
    [] (int, const std::vector<int>& xs) {
      return xs.empty();
    });

OK, passed 100 tests (2% trivial).

We can tag test cases too. The combination of tags on a test case form a category, and each category will be reported after the tests pass. The easiest way to tag a test case is to collect a value (that has an output stream operator):

cls.collect([] (int x, const std::vector<int>& xs) { return xs.size(); });

OK, passed 100 tests (2% trivial).
2% 0.
2% 1.
2% 10.
...

We can add tags for test cases that fit certain conditions:

cls.classify(
    [] (int x, const std::vector<int>& xs) {
       return xs.empty() || (x < xs.front());
    }, "at-head");        
cls.classify(
    [] (int x, const std::vector<int>& xs) {
      return xs.empty() || (xs.back() < x);
    }, "at-tail");         

OK, passed 100 tests (2% trivial).
2% 0, at-head, at-tail.
2% 10.
2% 12.
...