{"id":3045,"date":"2020-12-19T09:06:12","date_gmt":"2020-12-18T21:06:12","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=3045"},"modified":"2020-12-19T09:06:17","modified_gmt":"2020-12-18T21:06:17","slug":"big-updates-to-smoketest-2-x","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/3045\/","title":{"rendered":"Big Updates to Smoketest 2.x"},"content":{"rendered":"[Estimated Reading Time: <\/span> 5<\/span> minutes]<\/span><\/span>\n

A little over a year ago, I released Smoketest 2.0<\/a>, a complete re-write of the Smoketest unit testing framework first implemented some years ago. Other things then consumed my time, but in the past couple of months, Smoketest has rapidly progressed through no less than 4 (four!) fairly significant feature updates as I have extended it to support my testing needs in other projects.<\/p>\n\n\n\n

Today I released v2.4.0, and though it was about time (read: long overdue!)<\/em>, I shared some of the new features. As you might have gathered from my recent post on an exception handling bug in the Delphi 10.4.1 x86 compiler, exceptions have been a recent focus, so I’ll start with the changes in that area.<\/p>\n\n\n\n\n\n\n\n

\n\n\n\n

The 2.1.0 release was a big one, incorporating a major overhaul in the internals that greatly simplified some things; in particular, the support required for self-test extensions (for the framework to test itself) became easier to reason about. This was not something that affects the framework’s consumers but made my life easier and any contributor.<\/p>\n\n\n\n

For users<\/em> of the framework, there was an even bigger impact due to the introduction of a fluent Api for expressing tests. I’ll talk more about that in another post. For now, I’ll focus on the support in this fluent Api for an entirely new approach to testing for exceptions.<\/p>\n\n\n\n

To appreciate the significance of the test writing experience changes, it may help to recap the mechanism that it replaced.<\/p>\n\n\n\n

Previously to test for exceptions (in Smoketest 2.0), you would write something similar to the following:<\/p>\n\n\n\n

procedure TSomeTestClass.SomeTestMethod;\nbegin\n  try\n    SomeOperationUnderTest;\n\n    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');\n  except\n    AssertException( ESomeExceptionClass, 'The message text on the expected exception.');\n  end;\nend;<\/code><\/pre>\n\n\n\n
Two AssertException()<\/code> calls were required:  The first (in the try<\/code> block) would cause the test to fail since, for this statement to have been reached, the expected exception could not have been raised.  The exception specification was used to create the test failure report.<\/p>\n\n\n\n
The second (in the except<\/code> block) would test the exception in scope against the specification provided in the test and, if the current exception object matched that specification, the test would pass; otherwise, it would fail.<\/p>\n\n\n\n
There two immediately obvious problems with this:<\/p>\n\n\n\n
  1. The duplication is a clear violation of the DRY principle<\/a>.<\/li>
  2. Compounding that problem, the duplicated code behaved differently in the two contexts in which it was used<\/em>.  The first was expressly recording a failure, whereas the second was performing a test that could pass or fail.<\/li><\/ol>\n\n\n\n
    \n\n\n\n
    Eliminating Unnecessary<\/s> Duplication <\/h3>\n\n\n\n
    In 2.1.0, the need for the try..except boilerplate is eliminated, and the test is reduced to a single statement expressing the expectation that the test will raise an exception:<\/p>\n\n\n\n
    procedure TSomeTestClass.SomeTestMethod;\nbegin\n  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );\n\n  SomeOperationUnderTest;\nend;<\/code><\/pre>\n\n\n\n
    Without a try..except<\/code>, any exception raised by the method under test will be caught by the test framework, which then inspects the exception to verify that it matches the expected specification.<\/p>\n\n\n\n
    This is a massive<\/em> improvement in the test writing experience but has one major shortcoming: there can be only one such test in a given test method<\/em>.<\/p>\n\n\n\n
    In many cases, this is not a significant problem, but it is HUGE in those cases where it is a problem.<\/p>\n\n\n\n
    \n\n\n\n
    Once Is Not Always Enough<\/h3>\n\n\n\n
    Consider a scenario where some function is being tested to ensure that it raises an expected exception when presented with various invalid inputs.<\/p>\n\n\n\n
    A pattern I use for this is to implement a private<\/code> method in my test class, accepting data as input, and performing the required test.  The published<\/code>, parameterless test method itself then calls this private<\/code> method with the required input data:<\/p>\n\n\n\n
    \/\/ private\nprocedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);\nbegin\n  Test.RaisesException( ESomeExceptionClass, 'The message text on the expected exception.' );\n\n  SomeOperationUnderTest(aInput);\nend;\n\n\/\/ published\nprocedure TSomeTestClass.SomeTestMethod;\nbegin\n  SomeTestMethodWithData('foo');\n  SomeTestMethodWithData('bar');\nend;<\/code><\/pre>\n\n\n\n
    The problem here is that if the first call to SomeTestMethodWithData('foo') <\/code>does indeed raise the expected exception, the second call to SomeTestMethodWithData('bar')<\/code> will not be reached!<\/p>\n\n\n\n
    Ironically this was not a problem with the old mechanism, which relied on a try..except block to perform the exception tests within the test method’s scope.  Multiple tests could be performed in a series of try..except<\/code> blocks.<\/p>\n\n\n\n
    What was needed was some similar mechanism but which avoided the code duplication problem.  This is what has been developed for 2.4.0.<\/p>\n\n\n\n
    Let’s first look at what this looks like in use, then talk about how it works, using the ‘test with data’ scenario above to illustrate:<\/p>\n\n\n\n
    \/\/ private\nprocedure TSomeTestClass.SomeTestMethodWithData(const aInput: String);\nbegin\n  try\n    SomeOperationUnderTest(aInput);\n\n    Test.FailedToRaiseException;\n\n  except\n    Test.RaisedException( ESomeExceptionClass, 'The message text on the expected exception.' );\n  end;\nend;\n\n\/\/ published\nprocedure TSomeTestClass.SomeTestMethod;\nbegin\n  SomeTestMethodWithData('foo');\n  SomeTestMethodWithData('bar');\nend;<\/code><\/pre>\n\n\n\n
    The first thing to note is that this test is written in the past tense.<\/p>\n\n\n\n
    Rather than indicating an expectation of something that will happen in the future (Test.<strong><em>Raises<\/em><\/strong>Exception<\/code>), this test expresses something that is expected to have already happened: Test.<strong><em>Raised<\/em><\/strong>Exception<\/code><\/p>\n\n\n\n
    This is important because it captures the fact that the two tests are quite different in intent.  This allows internal validation that the correct test has been expressed.<\/p>\n\n\n\n
    NB. No such validation is implemented as yet, but is on the TODO list – “perfect” is the enemy of “done.”<\/em> \ud83d\ude42 <\/p>\n\n\n\n
    Since the exception test is entirely contained within a try..except<\/code> that does not “leak” any exception, all of the ..WithData()<\/code> tests are certain to be performed.<\/p>\n\n\n\n
    The outcome of the exception test in each call to those ..WithData()<\/code> methods is functionally equivalent to the 2.0 approach.<\/p>\n\n\n\n
    First, consider the “happy path” where the expected exception is raised.  An exception is caught by the try..except<\/code> and the Test.RaisedException()<\/code> call determines whether the raised exception matches the expected specification (the test passes) or not (the test fails).<\/p>\n\n\n\n
    So far, so straightforward.<\/p>\n\n\n\n
    If Test.FailedToRaiseException<\/code> is called, then this signifies that some expected exception was not<\/em><\/strong> raised, and so a test failure must be recorded.  But the framework does not know the particular exception that was expected to have been raised.  Simply recording “Some unspecified thing that was expected to have happened didn’t happen<\/em>” would not be very helpful!  What to do!?<\/p>\n\n\n\n
    Well, the required information is<\/em> available: in the test performed in response to an actual exception<\/em>.  Rather than burden the test author with duplicating the data, the framework goes and gets it for itself.<\/p>\n\n\n\n
    Internally the Test.FailedToRaiseException<\/code> method doesn’t actually record anything.  It does raise an exception of its own: an ENoException<\/code> exception, to be precise, defined internally by Smoketest.<\/p>\n\n\n\n
    This causes control to pass to the exception handler, which in turn results in Test.RaisedException()<\/code> being performed.  When this test finds that the exception raised was an ENoException<\/code>, it immediately<\/em> knows that the test has failed and records that fact using the exception specification it was provided with.<\/p>\n\n\n\n
    This is functionally identical to the old 2.0 mechanism but with clearer intent and without any need to duplicate details of the exception specification involved in the test!<\/p>\n\n\n\n
    \n\n\n\n
    Hammers and Screwdrivers<\/h3>\n\n\n\n
    Test.RaisedException<\/code> and Test.RaisesException<\/code> both exist and are supported by Smoketest 2.4.x.  They are not alternatives<\/em> to each other but satisfy different needs.<\/p>\n\n\n\n
    Where there is no need to perform multiple tests for exceptions, then Test.<strong>Raises<\/strong>Exception<\/code> can be used, avoiding the need for any try..except<\/code> boilerplate.<\/p>\n\n\n\n
    But where multiple<\/em> exception tests are required\/helpful, Test.FailedToRaiseException<\/code> and Test.<strong>Raised<\/strong>Exception<\/code> can be used with a try..except<\/code>.<\/p>\n\n\n\n
    \n\n\n\n
    Footnote<\/h2>\n\n\n\n
    In case you were wondering, this is the project that has consumed a lot of my time since September last year:<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Merry Christmas! \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"
    [Estimated Reading Time: <\/span>  5<\/span> minutes]<\/span><\/span>A little over a year ago, I released Smoketest 2.0, a complete re-write of the Smoketest unit testing framework first implemented some years ago. Other things then consumed my time, but in the past couple of months, Smoketest has rapidly progressed through no less than 4 (four!) fairly significant feature updates as I have extended […]<\/p>\n","protected":false},"author":2,"featured_media":2957,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Recent updates to Smoketest 2.x, post 1 of 'n'.  In this episode: Testing for expected exceptions.","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[4,321,348,48],"tags":[51],"class_list":["post-3045","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-delphi","category-devops","category-smoketest","category-testing","tag-smoketest"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/vape_2.jpg?fit=1229%2C314&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-N7","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=3045"}],"version-history":[{"count":7,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045\/revisions"}],"predecessor-version":[{"id":3053,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3045\/revisions\/3053"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media\/2957"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=3045"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=3045"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=3045"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}