{"id":3045,"date":"2020-12-19T09:06:12","date_gmt":"2020-12-18T21:06:12","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=3045"},"modified":"2020-12-19T09:06:17","modified_gmt":"2020-12-18T21:06:17","slug":"big-updates-to-smoketest-2-x","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/3045\/","title":{"rendered":"Big Updates to Smoketest 2.x"},"content":{"rendered":"[Estimated Reading Time: <\/span> 5<\/span> minutes]<\/span><\/span>\n

A little over a year ago, I released Smoketest 2.0<\/a>, a complete re-write of the Smoketest unit testing framework first implemented some years ago. Other things then consumed my time, but in the past couple of months, Smoketest has rapidly progressed through no less than 4 (four!) fairly significant feature updates as I have extended it to support my testing needs in other projects.<\/p>\n\n\n\n

Today I released v2.4.0, and though it was about time (read: long overdue!)<\/em>, I shared some of the new features. As you might have gathered from my recent post on an exception handling bug in the Delphi 10.4.1 x86 compiler, exceptions have been a recent focus, so I’ll start with the changes in that area.<\/p>\n\n\n\n\n\n\n\n

\n\n\n\n

The 2.1.0 release was a big one, incorporating a major overhaul in the internals that greatly simplified some things; in particular, the support required for self-test extensions (for the framework to test itself) became easier to reason about. This was not something that affects the framework’s consumers but made my life easier and any contributor.<\/p>\n\n\n\n

For users<\/em> of the framework, there was an even bigger impact due to the introduction of a fluent Api for expressing tests. I’ll talk more about that in another post. For now, I’ll focus on the support in this fluent Api for an entirely new approach to testing for exceptions.<\/p>\n\n\n\n

To appreciate the significance of the test writing experience changes, it may help to recap the mechanism that it replaced.<\/p>\n\n\n\n

Previously to test for exceptions (in Smoketest 2.0), you would write something similar to the following:<\/p>\n\n\n\n

procedure TSomeTestClass.SomeTestMethod;\nbegin\n  try\n    SomeOperationUnderTest;\n\n    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');\n  except\n    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');\n  end;\nend;<\/code><\/pre>\n\n\n\n
Two AssertException()<\/code> calls were required:  The first (in the try<\/code> block) would cause the test to fail since, for this statement to have been reached, the expected exception could not have been raised.  The exception specification was used to create the test failure report.<\/p>\n\n\n\n
The second (in the except<\/code> block) would test the exception in scope against the specification provided in the test and, if the current exception object matched that specification, the test would pass; otherwise, it would fail.<\/p>\n\n\n\n
There two immediately obvious problems with this:<\/p>\n\n\n\n
  1. The duplication is a clear violation of the DRY principle<\/a>.<\/li>
  2. Compounding that problem, the duplicated code behaved differently in the two contexts in which it was used<\/em>.  The first was expressly recording a failure, whereas the second was performing a test that could pass or fail.<\/li><\/ol>\n\n\n\n
    \n\n\n\n
    Eliminating Unnecessary<\/s> Duplication <\/h3>\n\n\n\n
    In 2.1.0, the need for the try..except boilerplate is eliminated, and the test is reduced to a single statement expressing the expectation that the test will raise an exception:<\/p>\n\n\n\n
    procedure TSomeTestClass.SomeTestMethod;\nbegin\n  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );\n\n  SomeOperationUnderTest;\nend;<\/code><\/pre>\n\n\n\n
    Without a try..except<\/code>, any exception raised by the method under test will be caught by the test framework, which then inspects the exception to verify that it matches the expected specification.<\/p>\n\n\n\n
    This is a massive<\/em> improvement in the test writing experience but has one major shortcoming: there can be only one such test in a given test method<\/em>.<\/p>\n\n\n\n
    In many cases, this is not a significant problem, but it is HUGE in those cases where it is a problem.<\/p>\n\n\n\n
    \n\n\n\n
    Once Is Not Always Enough<\/h3>\n\n\n\n
    Consider a scenario where some function is being tested to ensure that it raises an expected exception when presented with various invalid inputs.<\/p>\n\n\n\n
    A pattern I use for this is to implement a private<\/code> method in my test class, accepting data as input, and performing the required test.  The published<\/code>, parameterless test method itself then calls this private<\/code> method with the required input data:<\/p>\n\n\n\n
    \/\/ private\nprocedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);\nbegin\n  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );\n\n  SomeOperationUnderTest(aInput);\nend;\n\n\/\/ published\nprocedure TSomeTestClass.SomeTestMethod;\nbegin\n  SomeTestMethodWithData('foo');\n  SomeTestMethodWithData('bar');\nend;<\/code><\/pre>\n\n\n\n
    The problem here is that if the first call to SomeTestMethodWithData('foo') <\/code>does indeed raise the expected exception, the second call to SomeTestMethodWithData('bar')<\/code> will not be reached!<\/p>\n\n\n\n
    Ironically this was not a problem with the old mechanism, which relied on a try..except block to perform the exception tests within the test method’s scope.  Multiple tests could be performed in a series of try..except<\/code> blocks.<\/p>\n\n\n\n
    What was needed was some similar mechanism but which avoided the code duplication problem.  This is what has been developed for 2.4.0.<\/p>\n\n\n\n
    Let’s first look at what this looks like in use, then talk about how it works, using the ‘test with data’ scenario above to illustrate:<\/p>\n\n\n\n
    \/\/ private\nprocedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);\nbegin\n  try\n    SomeOperationUnderTest(aInput);\n\n    Test.FailedToRaiseException;\n\n  except\n    Test.RaisedException( ESomeExceptionClass, 'The message text on the expected exception.' );\n  end;\nend;\n\n\/\/ published\nprocedure TSomeTestClass.SomeTestMethod;\nbegin\n  SomeTestMethodWithData('foo');\n  SomeTestMethodWithData('bar');\nend;<\/code><\/pre>\n\n\n\n
    The first thing to note is that this test is written in the past tense.<\/p>\n\n\n\n
    Rather than indicating an expectation of something that will happen in the future (Test.<strong><em>Raises<\/em><\/strong>Exception<\/code>), this test expresses something that is expected to have already happened: Test.<strong><em>Raised<\/em><\/strong>Exception<\/code><\/p>\n\n\n\n
    This is important because it captures the fact that the two tests are quite different in intent.  This allows internal validation that the correct test has been expressed.<\/p>\n\n\n\n
    NB. No such validation is implemented as yet, but is on the TODO list – “perfect” is the enemy of “done.”<\/em> \ud83d\ude42 <\/p>\n\n\n\n
    Since the exception test is entirely contained within a try..except<\/code> that does not “leak” any exception, all of the ..WithData()<\/code> tests are certain to be performed.<\/p>\n\n\n\n
    The outcome of the exception test in each call to those ..WithData()<\/code> methods is functionally equivalent to the 2.0 approach.<\/p>\n\n\n\n
    First, consider the “happy path” where the expected exception is raised.  An exception is caught by the try..except<\/code> and the Test.RaisedException()<\/code> call determines whether the raised exception matches the expected specification (the test passes) or not (the test fails).<\/p>\n\n\n\n
    So far, so straightforward.<\/p>\n\n\n\n
    If Test.FailedToRaiseException<\/code> is called, then this signifies that some expected exception was not<\/em><\/strong> raised, and so a test failure must be recorded.  But the framework does not know the particular exception that was expected to have been raised.  Simply recording “Some unspecified thing that was expected to have happened didn’t happen<\/em>” would not be very helpful!  What to do!?<\/p>\n\n\n\n
    Well, the required information is<\/em> available: in the test performed in response to an actual exception<\/em>.  Rather than burden the test author with duplicating the data, the framework goes and gets it for itself.<\/p>\n\n\n\n
    Internally the Test.FailedToRaiseException<\/code> method doesn’t actually record anything.  It does raise an exception of its own: an ENoException<\/code> exception, to be precise, defined internally by Smoketest.<\/p>\n\n\n\n
    This causes control to pass to the exception handler, which in turn results in Test.RaisedException()<\/code> being performed.  When this test finds that the exception raised was an ENoException<\/code>, it immediately<\/em> knows that the test has failed and records that fact using the exception specification it was provided with.<\/p>\n\n\n\n
    This is functionally identical to the old 2.0 mechanism but with clearer intent and without any need to duplicate details of the exception specification involved in the test!<\/p>\n\n\n\n
    \n\n\n\n
    Hammers and Screwdrivers<\/h3>\n\n\n\n
    Test.RaisedException<\/code> and Test.RaisesException<\/code> both exist and are supported by Smoketest 2.4.x.  They are not alternatives<\/em> to each other but satisfy different needs.<\/p>\n\n\n\n
    Where there is no need to perform multiple tests for exceptions, then Test.<strong>Raises<\/strong>Exception<\/code> can be used, avoiding the need for any try..except<\/code> boilerplate.<\/p>\n\n\n\n
    But where multiple<\/em> exception tests are required\/helpful, Test.FailedToRaiseException<\/code> and Test.<strong>Raised<\/strong>Exception<\/code> can be used with a try..except<\/code>.<\/p>\n\n\n\n
    \n\n\n\n
    Footnote<\/h2>\n\n\n\n
    In case you were wondering, this is the project that has consumed a lot of my time since September last year:<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Merry Christmas! \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"
    [Estimated Reading Time: <\/span> 5<\/span> minutes]<\/span><\/span> A little over a year ago, I released Smoketest 2.0, a complete re-write of the Smoketest unit testing framework first implemented some years ago. Other things then consumed my time, but in the past couple of months, Smoketest has rapidly progressed through no less than 4 (four!) fairly significant feature updates as I have extended […]<\/p>\n","protected":false},"author":2,"featured_media":2957,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"Recent updates to Smoketest 2.x, post 1 of 'n'.  In this episode: Testing for expected exceptions.","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[4,321,348,48],"tags":[51],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/vape_2.jpg?fit=1229%2C314&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-N7","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2185,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2185\/","url_meta":{"origin":3045,"position":0},"title":"Smoketest – Performance Case Visualisations","date":"22 Nov 2013","format":false,"excerpt":"This post is a peek behind the curtain of the next major update to Smoketest which I hope to have completed shortly: Performance Case visualisations. Smoketest has always had two types of test case that you could implement by deriving from two distinct base classes: TTestCase is the base class\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/Screen-Shot-2013-11-22-at-09.57.52.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2144,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2144\/","url_meta":{"origin":3045,"position":1},"title":"Smoketest 1.0 – Release and Be Damned!","date":"14 Nov 2013","format":false,"excerpt":"As I have been promising for some time (quite literally 5 years (!), I am ashamed to admit) I am finally unclenching and releasing the Smoketest framework into the wild, ready or not. The code is published and will continue to be updated in a github repository. Documentation is still\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/Screen-Shot-2013-11-14-at-19.51.57-.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2095,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2095\/","url_meta":{"origin":3045,"position":2},"title":"Extending Smoketest (Part 1) – An Inspector Calls","date":"05 Nov 2013","format":false,"excerpt":"In the soon to be released Smoketest framework it is sometimes useful to create new test types to supplement the tests built-in to the framework. In this and the next post I will walk through the process of implementing and registering a custom test with the Smoketest framework. In a\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":284,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/284\/","url_meta":{"origin":3045,"position":3},"title":"This Week’s Poll, Delphi at Tech Ed and 2009 Ship Date","date":"01 Sep 2008","format":false,"excerpt":"Since I'm preparing a series of posts about (and eventual publication of) Smoketest, my own testing framework, I thought I'd test the water with this weeks poll and see what - if anything - people are already using as far as unit testing goes. Also I thought I'd briefly mention\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2956,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2956\/","url_meta":{"origin":3045,"position":4},"title":"Introducing: Smoketest 2.0","date":"24 Sep 2019","format":false,"excerpt":"Meet the new Smoketest. Nothing like the old Smoketest. A new, ultra-lightweight unit testing framework with xUnit 2.x result output (and the ability to support other output formats as needed).","rel":"","context":"In "automation"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/vape_2.jpg?fit=1200%2C307&ssl=1&resize=350%2C200","width":350,"height":200},"classes":[]},{"id":2164,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2164\/","url_meta":{"origin":3045,"position":5},"title":"Smoketest – Some Differences With DUnit","date":"19 Nov 2013","format":false,"excerpt":"Writing tests in Smoketest is intended to enable a test developer to write tests in a way that describe themselves, without requiring the test developer to add this \"narrative\" themselves. To see this in action, I thought I would compare some simple DUnit tests with the equivalent using the Smoketest\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045"}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=3045"}],"version-history":[{"count":7,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045\/revisions"}],"predecessor-version":[{"id":3053,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045\/revisions\/3053"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media\/2957"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=3045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=3045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=3045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}