Lately I was busy adding tests that are able to verify if some code (e.g. an optimizer that uses random, like genetic algorithm, simulated annealing, …) produces the desired in e.g. 95% of the cases (Wikipedia has a nice practical example for confidence intervals).

Example Usage

Here is an example that tests the nonsense code Math.random() * 2.

ensure(new Experiment() {
    public double measure() {
        return Math.random() * 2;
    }
}).between(0.9, 1.1, 0.95).sample(10, 100);

The code most likely does not make much sense out of context like this, so here is an explanation of what it does:

Explanation

In that example we want to verify that the code returns values whose 95% confidence interval is between 0.9 and 1.1. At first take 10 samples to verify this. If the mean is not as expected between the interval, there is still the chance that this was just bad luck. We can take further samples (up to 1000) to rule bad luck out. When we have to take more than 1000 samples and the mean is still not in the specified still not met, there is a very high probability that the code does not produce the desired result, so we have to fail.

I hope the above description is somewhat understandable. It’s no piece of cake, and there is no way to cheat around the complexity involved with that kind of statistical tests. I have tried to make the interface as clean and simple as possible, suggestions are always welcome.

Open Source?

I am developing ensure4j partly at work, and partly in my free time. I would like to make it open source, but I need to get the OK from my boss first.

ensure4j is the the only implementation I know of that uses statistical tests that can be used with JUnit. If you know something similar than this, please post!

This are the goals I have for the redesign:

• Make the code expressive and as readable as possbile. This means the code should look like plain english.
• Usability is more important than flexibility.
• Allow for much more poweful checks.

Usability is very important. The good thing is that it is easy to analyze how JUnit is used (download some test driven developed open source projects, grep for assertTrue, and there you have all the cases where JUnit’s asserts were not good enough )

I am playing around with a few ideas, and this is the API I am currently after:

Assertions Example Usage

public void testInt() {
  int num = 4+1;
  ensure(num).is(5);
  ensure(num).either(4, 5, 6);
  ensure(num).between(3, 10);
}

public void testDouble() {
  double val = 4 + 0.4;
  // an epsilon is mandatory when comparing double
  ensure(val).is(4.4, 0.0001);
  ensure(val).between(4.3, 4.5);
}

public void testArray() {
  int[] a = { 1, 2, 3, 4, 5 };
  ensure(a).contains(3);
  ensure(a).size().either(5, 6);
  ensure(a).contains(2, 3, 4);
  ensure(a).contains().either(3, 10, 11);
  ensure(a).contains().neither(6, 7, 9);
  ensure(a).contains().any(3, 4);
  ensure(a).contains().all(3, 4, 5);
  ensure(a).isSorted();
  ensure(a).isUnique();
}

Starting point is always ensure(). You put in the actual value, then execute some operations that modify the following constraint check, then the last command is the constraint check. The last command is always the constraint check, so you can only do one check per line of code. That’s not as flexible as jMock, but it is simpler and more readable and good enough.

Technically all this is very doable. Probably quite a bit of work, and most likely requires a code generator since lots of code is almost the same but not reusable (e.g. arrays for primitive data types), but that’s not a problem. I have done a few simple feasibility studies and I don’t see a big problem with this interface.

Please tell me what you think about this!

Updates

From time to time I will update this page with progress reports, until I have something that is stable enough to give away.

2006-08-05

I have started developing this. So far I am very happy with it, its a fun project to implement I will probably create a project on sourceforge and opensource it, but I am not sure about the licence.