[Estimated Reading Time: 5 minutes]

A little over a year ago, I released Smoketest 2.0, a complete re-write of the Smoketest unit testing framework first implemented some years ago. Other things then consumed my time, but in the past couple of months, Smoketest has rapidly progressed through no less than 4 (four!) fairly significant feature updates as I have extended it to support my testing needs in other projects.

Today I released v2.4.0, and though it was about time (read: long overdue!), I shared some of the new features. As you might have gathered from my recent post on an exception handling bug in the Delphi 10.4.1 x86 compiler, exceptions have been a recent focus, so I’ll start with the changes in that area.

The 2.1.0 release was a big one, incorporating a major overhaul in the internals that greatly simplified some things; in particular, the support required for self-test extensions (for the framework to test itself) became easier to reason about. This was not something that affects the framework’s consumers but made my life easier and any contributor.

For users of the framework, there was an even bigger impact due to the introduction of a fluent Api for expressing tests. I’ll talk more about that in another post. For now, I’ll focus on the support in this fluent Api for an entirely new approach to testing for exceptions.

To appreciate the significance of the test writing experience changes, it may help to recap the mechanism that it replaced.

Previously to test for exceptions (in Smoketest 2.0), you would write something similar to the following:

procedure TSomeTestClass.SomeTestMethod;
begin
  try
    SomeOperationUnderTest;

    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');
  except
    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');
  end;
end;

Two AssertException() calls were required: The first (in the try block) would cause the test to fail since, for this statement to have been reached, the expected exception could not have been raised. The exception specification was used to create the test failure report.

The second (in the except block) would test the exception in scope against the specification provided in the test and, if the current exception object matched that specification, the test would pass; otherwise, it would fail.

There two immediately obvious problems with this:

  1. The duplication is a clear violation of the DRY principle.
  2. Compounding that problem, the duplicated code behaved differently in the two contexts in which it was used. The first was expressly recording a failure, whereas the second was performing a test that could pass or fail.

Eliminating Unnecessary Duplication

In 2.1.0, the need for the try..except boilerplate is eliminated, and the test is reduced to a single statement expressing the expectation that the test will raise an exception:

procedure TSomeTestClass.SomeTestMethod;
begin
  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );

  SomeOperationUnderTest;
end;

Without a try..except, any exception raised by the method under test will be caught by the test framework, which then inspects the exception to verify that it matches the expected specification.

This is a massive improvement in the test writing experience but has one major shortcoming: there can be only one such test in a given test method.

In many cases, this is not a significant problem, but it is HUGE in those cases where it is a problem.

Once Is Not Always Enough

Consider a scenario where some function is being tested to ensure that it raises an expected exception when presented with various invalid inputs.

A pattern I use for this is to implement a private method in my test class, accepting data as input, and performing the required test. The published, parameterless test method itself then calls this private method with the required input data:

// private
procedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);
begin
  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );

  SomeOperationUnderTest(aInput);
end;

// published
procedure TSomeTestClass.SomeTestMethod;
begin
  SomeTestMethodWithData('foo');
  SomeTestMethodWithData('bar');
end;

The problem here is that if the first call to SomeTestMethodWithData('foo') does indeed raise the expected exception, the second call to SomeTestMethodWithData('bar') will not be reached!

Ironically this was not a problem with the old mechanism, which relied on a try..except block to perform the exception tests within the test method’s scope. Multiple tests could be performed in a series of try..except blocks.

What was needed was some similar mechanism but which avoided the code duplication problem. This is what has been developed for 2.4.0.

Let’s first look at what this looks like in use, then talk about how it works, using the ‘test with data’ scenario above to illustrate:

// private
procedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);
begin
  try
    SomeOperationUnderTest(aInput);

    Test.FailedToRaiseException;

  except
    Test.RaisedException( ESomeExceptionClass, 'The message text on the expected exception.' );
  end;
end;

// published
procedure TSomeTestClass.SomeTestMethod;
begin
  SomeTestMethodWithData('foo');
  SomeTestMethodWithData('bar');
end;

The first thing to note is that this test is written in the past tense.

Rather than indicating an expectation of something that will happen in the future (Test.<strong><em>Raises</em></strong>Exception), this test expresses something that is expected to have already happened: Test.<strong><em>Raised</em></strong>Exception

This is important because it captures the fact that the two tests are quite different in intent. This allows internal validation that the correct test has been expressed.

NB. No such validation is implemented as yet, but is on the TODO list – “perfect” is the enemy of “done.” 🙂

Since the exception test is entirely contained within a try..except that does not “leak” any exception, all of the ..WithData() tests are certain to be performed.

The outcome of the exception test in each call to those ..WithData() methods is functionally equivalent to the 2.0 approach.

First, consider the “happy path” where the expected exception is raised. An exception is caught by the try..except and the Test.RaisedException() call determines whether the raised exception matches the expected specification (the test passes) or not (the test fails).

So far, so straightforward.

If Test.FailedToRaiseException is called, then this signifies that some expected exception was not raised, and so a test failure must be recorded. But the framework does not know the particular exception that was expected to have been raised. Simply recording “Some unspecified thing that was expected to have happened didn’t happen” would not be very helpful! What to do!?

Well, the required information is available: in the test performed in response to an actual exception. Rather than burden the test author with duplicating the data, the framework goes and gets it for itself.

Internally the Test.FailedToRaiseException method doesn’t actually record anything. It does raise an exception of its own: an ENoException exception, to be precise, defined internally by Smoketest.

This causes control to pass to the exception handler, which in turn results in Test.RaisedException() being performed. When this test finds that the exception raised was an ENoException, it immediately knows that the test has failed and records that fact using the exception specification it was provided with.

This is functionally identical to the old 2.0 mechanism but with clearer intent and without any need to duplicate details of the exception specification involved in the test!

Hammers and Screwdrivers

Test.RaisedException and Test.RaisesException both exist and are supported by Smoketest 2.4.x. They are not alternatives to each other but satisfy different needs.

Where there is no need to perform multiple tests for exceptions, then Test.<strong>Raises</strong>Exception can be used, avoiding the need for any try..except boilerplate.

But where multiple exception tests are required/helpful, Test.FailedToRaiseException and Test.<strong>Raised</strong>Exception can be used with a try..except.

Footnote

In case you were wondering, this is the project that has consumed a lot of my time since September last year:

Merry Christmas! 🙂