{"id":3009,"date":"2019-10-10T09:13:15","date_gmt":"2019-10-09T21:13:15","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=3009"},"modified":"2019-10-10T09:13:15","modified_gmt":"2019-10-09T21:13:15","slug":"smoketest-2-0-3-update","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/3009\/","title":{"rendered":"Smoketest 2.0.3 Update"},"content":{"rendered":"[Estimated Reading Time: <\/span> 5<\/span> minutes]<\/span><\/span>\n

Some minor improvements to the Smoketest<\/strong> framework went live earlier this week. Some cosmetic, some functional.<\/p>\n\n\n\n\n\n\n\n

More than Skin Deep<\/h4>\n\n\n\n

The most subtle change in this release is in the form of additional documentation added to the comments in the implementation itself.<\/p>\n\n\n\n

I hope people find that useful. \ud83d\ude42<\/p>\n\n\n\n

Just a Prettier Face<\/h4>\n\n\n\n

On the cosmetic front, console output from the test run is now improved [subjective opinion only, ymmv – Ed<\/em>].<\/p>\n\n\n\n

The first thing taken care of was the display of tests outcomes with overridden expectations. This is something that is only relevant to the special case of the self-tests provided with the framework but now means that when a self-test is expected to fail, if it correctly fails then this is now reported as a PASS in the console output. Correct outcomes were already taken care of in the results writers, so this was just an inconsistency that needed tidying up. <\/p>\n\n\n\n

In more general changes, output from tests in different test classes are now separated visually by a single blank line and each test class is introduced. Test outcomes recorded by the methods in that class are then output with a leading +<\/code> resulting in a pleasing indentation of those test results relative to the containing class.<\/p>\n\n\n\n

Any warnings output by the test run are not indented or prefixed helping these to stand out from the test outcomes.<\/p>\n\n\n\n

One of the warnings now output is in the case of a test method that performs no tests. That is, no test results are recorded during the execution of that method.<\/p>\n\n\n\n

WARNING: Test method <className>.<methodName> did not perform any tests<\/code><\/p>\n\n\n\n

A similar warning is emitted if a test class fails to implement any<\/em> test methods at all:<\/p>\n\n\n\n

WARNING: Test class <className> implements no test methods<\/code><\/p>\n\n\n\n

In both cases the warning has no impact on any test results written by any writer and in fact are not even detectable to a results writer. They are emitted solely by the test runner.<\/p>\n\n\n\n

For the first of these methods a mechanism is provided to suppress this warning if required. Which brings us to the functional changes.<\/p>\n\n\n\n

NOT Just a Prettier Face<\/h4>\n\n\n\n

Any method that deliberately and knowingly does not perform any test methods can call TestRun.PerformsNoTests<\/code> to suppress the no test methods<\/code> warning.<\/p>\n\n\n\n

Why would you do this ?<\/p>\n\n\n\n

I provided this as I ran into a scenario when expanding the self-tests where I wanted to perform some scaffolding of some test data before subsequents test methods were executed. Rather than overloading a test method with this scaffolding in addition to performing an initial test, I decided that such things should be handled specifically by scaffolding methods.<\/p>\n\n\n\n

Such methods still need to be published as test methods for the test runner to be able to discover and execute them.<\/p>\n\n\n\n

The specific example, if you are interested, is the CaptureStats<\/code> method of the TCoreFunctionalityTests<\/code> in the tests\\selftest<\/code> project.<\/p>\n\n\n\n

Currently this sort of thing relies on predictable order of execution based on the declared order of methods in the test class declaration. This makes me nervous and may be revisited to find a more robust solution but does the job that is needed for now.<\/p>\n\n\n\n

A bigger, but somewhat related, change is in the fact that Setup<\/strong> and Teardown<\/strong> methods are now supported.<\/p>\n\n\n\n

\n\n\n\n

Set Me Up and Tear Me Down…<\/h4>\n\n\n\n

Setup and Teardown relies on test methods in a class having special names. Test methods with these names are removed from the list of test methods executed as part of the test class and are instead invoked at specific points during test execution for that class:<\/p>\n\n\n\n

Method Name       When The Method Is Called\n---------------   -------------------------------------------------\nSetupTest         ONCE per test class run, before any test methods\n                   in the class\n\nSetupMethod       ONCE PER TEST METHOD, before each and every\n                   test method in the class\n\nTeardownMethod    ONCE PER TEST METHOD, after each and\n                   every test method in the class\n\nTeardownTest      ONCE per test class run, after all test methods\n                   in a class<\/pre>\n\n\n\n
These methods do not have to be ‘paired up’.  You can have Setup<\/strong> methods without corresponding Teardown<\/strong> methods, and\/or vice-versa.  When any of these methods is present a line is output to the console to indicate their execution.<\/p>\n\n\n\n
There is nothing currently to stop you from writing tests in these methods, it’s just not a good idea to do so.  It would be possible to abend a test run if test results were detected from these methods and this may be added in the future, or at least a WARNING emitted when this occurs. <\/p>\n\n\n\n
It also doesn’t matter where these methods are declared, relative to any other test methods.  They do still need to be published<\/code> methods but can appear anywhere in the list of such methods.  As long as they are named with the correct names as described above, they will be identified and invoked by the test runner at the appropriate points, automatically.<\/p>\n\n\n\n
NOTE:<\/strong> If an exception escapes from any of these methods it will be caught and logged to the console but the test runner will then carry on with the test run as normal.<\/p>\n\n\n\n
For an example of these methods in action you can again consult the self tests, specifically in this case the TCommandLineHandlingTests<\/code> in Test.CommandLineHandling<\/code>.<\/p>\n\n\n\n
For some idea of the impact of these changes, here’s a capture of a test run of those self tests:<\/p>\n\n\n\n
Smoketest test run: Name=selftest.exe, Environment=Delphi XE4\nWriters supported: xunit2\n\nExecuting tests in TCoreFunctionalityTests:\n+ This test passed: PASSED\n+ This test failed: PASSED\n+ ThisTestWillThrowAnException test 1: PASSED\n+ ThisTestWillThrowAnException test 1 threw expected exception [Exception: This exception was deliberately raised]\n+ 3 tests recorded to this point: PASSED\n+ 1 test passed at this point: PASSED\n+ 1 test failed at this point: PASSED\n+ 1 test error at this point: PASSED \n\nExecuting tests in TExceptionHandlingTests:\n+ EDivByZero caught by AssertException(EDivByZero): PASSED\n+ EDivByZero caught by AssertBaseException(Exception): PASSED\n+ EDivByZero not caught by Assert(Exception) causes test to fail: PASSED\n+ Unexpected Exception raised causes test to fail: PASSED \n\nExecuting tests in TCommandLineHandlingTests:\nPerforming setup for test\nPerforming setup for method:SwitchPresentWithNoValueIsHandledCorrectly\n+ Present -switch is identified: PASSED\n+ Present -switch has no value: PASSED\nPerforming setup for method:SwitchPresentWithValueIsHandledCorrectly\n+ -mode switch is identified: PASSED\n+ -mode value is 'level=42': PASSED\nPerforming setup for method:SwitchNotPresentIsHandledCorrectly\n+ Missing -lever is not identified: PASSED\n+ Missing -lever has no value: PASSED\nPerforming setup for method:QuotedValueIsUnquotedCorrectly\n+ Quoted value is unquoted correctly: PASSED\n+ Performing teardown for test \n\nTotal Tests = 18, Passed = 18, Failed = 0, Skipped = 0, Errors = 0<\/code><\/pre>\n\n\n\n
All of these changes have no impact on the results writers which continue to function as before.<\/p>\n\n\n\n
One small change which has<\/em> had a minor impact on results writers is that the Runtime<\/code> property on the test run, which records the elapsed time of the overall test run.  This is now reported in milliseconds rather than seconds.<\/p>\n\n\n\n
The existing xUnit2<\/strong> results writer has already been updated to reflect this and so provide more ‘accurate’ timings in the results that it produces.<\/p>\n\n\n\n
\n\n\n\n
More To Come<\/h4>\n\n\n\n
As part of the implementation of duget<\/strong> I am going through my existing libraries of code that I rely on extensively, re-organising them for packaging with duget<\/strong>.  Part of this involves migrating the tests to Smoketest 2.0<\/strong> (or adding them where not already present).<\/p>\n\n\n\n
This is driving further improvements to Smoketest 2.0<\/strong>, with a lot of work at the moment in the Api for expressing Assert<\/strong>s but it’s all quite fluid at the moment.<\/p>\n\n\n\n
This will be a significant addition to the framework and I anticipate a bump to 2.1.0 once things have settled down and I have a clear direction for the future evolution of this aspect of the framework.<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span>  5<\/span> minutes]<\/span><\/span>An overview of the changes in Smoketest 2.0.3 including Setup\/Teardown methods and improved formatting of console output.<\/p>\n","protected":false},"author":2,"featured_media":2957,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Overview of the Smoketest 2.0.3 Update - In Summary: Setup\/Teardown methods and improved formatting of console output.","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[323,4,321,348,48],"tags":[327,329,51,349,295],"class_list":["post-3009","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ci-cd","category-delphi","category-devops","category-smoketest","category-testing","tag-ci-cd","tag-devops","tag-smoketest","tag-tdd","tag-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/vape_2.jpg?fit=1229%2C314&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-Mx","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3009","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=3009"}],"version-history":[{"count":3,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3009\/revisions"}],"predecessor-version":[{"id":3012,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/3009\/revisions\/3012"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media\/2957"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=3009"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=3009"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=3009"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}