{"id":2956,"date":"2019-09-24T09:18:28","date_gmt":"2019-09-23T21:18:28","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=2956"},"modified":"2019-09-24T09:18:36","modified_gmt":"2019-09-23T21:18:36","slug":"introducing-smoketest-2-0","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/2956\/","title":{"rendered":"Introducing: Smoketest 2.0"},"content":{"rendered":"[Estimated Reading Time: <\/span> 6<\/span> minutes]<\/span><\/span>\n

Many years ago I created Smoketest<\/a> and had a lot of fun doing so. But recently I ran into a problem when trying to use it for creating tests for my code that I am currently preparing for packaging. The problem was that it was just too darned ‘heavy’.<\/p>\n\n\n\n

In particular, it made extensive use of the very code I was trying to test with it. In the past I admit I had just lived with this, developing all sorts of coping strategies. But in trying to automate my CI\/CD pipelines these headaches grew to migraine levels and enough was enough.<\/p>\n\n\n\n

Initially I set about creating a much more basic and crude set of tests for my lowest level libraries, thinking that if I could just get test coverage in place at that level then once I had all the dependencies that Smoketest<\/strong> needed I could then start using that for those higher-level, more complex testing needs.<\/p>\n\n\n\n

Because of the code it was needed to test, this basic testing approach needed to be dependency-free, relying solely on the Delphi RTL<\/strong> and being portable across all<\/em> Delphi<\/strong> versions from 7 to 10.3<\/strong>. These constraints actually made the job easier as it limited options and kept things very basic indeed.<\/p>\n\n\n\n

In fact, I had something that did this job in a little over half a day, even with support for outputting results in xUnit 2<\/strong> format! I then realised that what I had come up with needed only a little more work to become the basis for what I have decided will be Smoketest 2.0<\/a><\/strong>, available now on GitHub<\/a>.<\/p>\n\n\n\n

\n\n\n\n

Weighing in at a smidgen over 40KB of code, Smoketest 2.0<\/strong> is an absolute featherweight compared to the original at 450KB+<\/strong><\/em> of code. In fact, the original is actually embarrassingly bloated, topping the scales at a whopping 2-3 MEGABytes<\/strong> with all the resources for the built-in UI runner taken into account! I know that this isn’t necessarily important or even significant, but it’s an interesting data-point none-the-less.<\/p>\n\n\n\n

The first thing to get out of the way is that Smoketest 2.0<\/strong> has absolutely nothing in common with the previous Smoketest<\/strong> framework other than the name. Any existing Smoketest<\/strong> tests you have will need to be re-purposed. I have the same problem.<\/p>\n\n\n\n

Being so completely different, I wrestled with whether to keep the name and initially had a new name in mind. Eventually however, I concluded that this was the genesis of a total Smoketest<\/strong> replacement, and hence a completely new version of Smoketest<\/strong> rather than something actually separate and new.<\/p>\n\n\n\n

Everything you really need to know is in the README.md<\/a><\/strong> for the project on GitHub, though the Getting Started<\/strong> section is a bit of a tease. It references duget<\/strong> which has yet to be revealed to the world (coming soon, I promise).<\/p>\n\n\n\n

For now, these instructions don’t make much sense but fortunately because of the dependency-free nature of Smoketest 2.0<\/strong>, using it is actually super-easy. Barely and inconvenience [You really should stop using that or Ryan George will on your back! – Ed<\/em>]<\/p>\n\n\n\n

The REAL ‘Getting Started’ Guide<\/h3>\n\n\n\n

Clone the repo somewhere convenient and add the src<\/code> folder to the path of any project (or globally). I use scope resolving declarations to elevate types<\/strong> and consts<\/strong> etc throughout the framework where needed, so:<\/p>\n\n\n\n

To write a test suite you need only add Deltics.Smoketest<\/code> to your program<\/code> uses<\/code> clause and call the appropriate methods on the TestRun<\/code> object that this introduces. At a minimum this means making at least one call to TestRun.Test<\/code>, passing a test class (or array of test classes) to be executed.<\/p>\n\n\n\n

To implement a test class that you can pass to the TestRun<\/code> you derive a class from the TTest<\/code> class. Again, this needs only Deltics.Smoketest<\/code> present in your uses<\/code> clause. The tests performed by a test class will be any published, parameter-less methods.<\/p>\n\n\n\n

To write tests in your test methods, use the Assert<\/code> method inherited from the TTest<\/code> class. AssertException<\/code> and AssertBaseException<\/code> are also provided for super-advanced exception handling tests.<\/p>\n\n\n\n

Examples for all three of these are provided in the README<\/a><\/strong>.<\/p>\n\n\n\n

Output of results to file is handled automatically by Smoketest<\/strong> upon completion of the test run, if the appropriate format option is provided on the command line. Different output formats are supported by implementations of a ResultsWriter<\/strong> class. Each class is registered in the Smoketest framework with a corresponding format identifier. To output in a particular format you simply specify a command line option to the test run where the option has the name of the format and has a value which identifies a filename to hold the results in that format.<\/p>\n\n\n\n

At time of writing only one format is supported: XUnit 2.x. The format identifier for this is xunit2<\/code> so to emit test results in this format you would run your test suite with a command line similar to:<\/p>\n\n\n\n

mytests -xunit2:test-results.xml<\/code><\/p>\n\n\n\n

The xUnit2<\/strong> results writer supports appending results. If the filename specified already exists then the writer will assume that you wish to add the results of this run to those already in the file and will add them as a new <assembly><\/code>. Whether this makes sense may vary from test suite to test suite but if you want a separate file for each test run results then you will need to contrive some way of specifying unique filenames to your test runs.<\/p>\n\n\n\n

\n\n\n\n

Below is an example of a test suite taken from one of the packages I am currently, uh, packaging. It simply packages up the include file that I use across all my projects for detecting and adapting to various Delphi versions. This test suite is a little unusual therefore in that the tests are testing $DEFINE<\/code> symbols and the only ‘code’ under test is the include file.<\/p>\n\n\n\n

First, the test suite:<\/p>\n\n\n

\n{$apptype CONSOLE}\n\n{$i deltics.inc}\n\nprogram Test;\n\n  uses\n    Deltics.Smoketest,\n    TestConsts in 'TestConsts.pas',\n    TestCore in 'TestCore.pas',\n    TestVersionDefines in 'TestVersionDefines.pas';\n\nbegin\n  TestRun.Environment := 'Delphi ' + Uppercase(DELPHI_VERSION);\n  try\n    TestRun.Test(TCoreFunctionality);\n    TestRun.Test(TVersionDefines, UpperCase(DELPHI_VERSION));\n\n  except\n    on e: Exception do\n      WriteLn('ERROR: ' + e.Message);\n  end;\nend.\n<\/pre><\/div>\n\n\n
The Environment<\/strong> property which is set is used in the test result output.  A  Name<\/strong> property can also be set.  Both are optional and default values will be used if not provided.  Name<\/strong> will default to the executable filename and Environment<\/strong> will default to ‘Test’.<\/p>\n\n\n\n
These values are fixed once the first test has been performed in a run.<\/p>\n\n\n\n
Then the suite runs two test classes.  The second is run with a NamePrefix<\/strong> parameter.  This is a prefix that is prefixed to every test name produced by that test.  This is for discriminating between results where the same names are used for different runs of the same tests.  In this case, this test suite will be executed for multiple Delphi versions (separate compilations and executions), so the Delphi version is a useful prefix.<\/p>\n\n\n\n
As well as making it easier for humans to interpret results, this enables the test tracking performed by Azure DevOps to identify when a particular test that was broken has now become fixed.<\/p>\n\n\n\n
For the tests themselves, here’s an example from the same test suite:<\/p>\n\n\n
\n  unit TestCore;\n\n{$i deltics.inc}\n\ninterface\n\n  uses\n    Deltics.Smoketest;\n\n  type\n    TCoreFunctionality = class(TTest)\n      procedure DelphiVersionIsIdentified;\n      procedure DelphiVersionIsTheExpectedVersion;\n    end;\n\n\n\nimplementation\n\n  uses\n    SysUtils,\n    TestConsts;\n\n\n  procedure TCoreFunctionality.DelphiVersionIsIdentified;\n  begin\n    Assert('UNKNOWN_COMPILER_VERSION is not defined',\n           {$ifdef UNKNOWN_COMPILER_VERSION} FALSE {$else} TRUE {$endif},\n           'UNKNOWN_COMPILER_VERSION *is* defined!');\n  end;\n\n\n  procedure TCoreFunctionality.DelphiVersionIsTheExpectedVersion;\n  var\n    expectedVersion: String;\n  begin\n    if NOT TestRun.HasCmdLineOption('delphiVersion', expectedVersion) then\n    begin\n      Console.WriteLn('ERROR: -delphiVersion:<version> command line parameter missing');\n      TestRun.Abort;\n    end;\n\n    if NOT Assert('Expected version = Detected version',\n                  SameText(expectedVersion, DELPHI_VERSION),\n                  Format('Expected version ''%s'', Detected version %s', [expectedVersion, DELPHI_VERSION])) then\n      TestRun.Abort;\n  end;\n\n\nend.\n<\/pre><\/div>\n\n\n
This is a slightly clunky example since what is being tested here are the settings of compiler $DEFINE<\/code><\/strong> symbols, but the principle should be clear.<\/p>\n\n\n\n
For a test, you write an Assert()<\/code><\/strong> that has at least two parameters.  The first is the name of the test that the assert applies.  This is used in the test output to identify the individual test.<\/p>\n\n\n\n
The second parameter is a boolean – if this is TRUE<\/strong> then the test is considered to have passed.  If FALSE<\/strong> then it failed.<\/p>\n\n\n\n
The third parameter is optional and may be specified to provide a  message that will be attached to the test result.  In most cases results writers will output this message only for tests that fail or end in an error state.<\/p>\n\n\n\n
In the example above, the second test method a demonstrates a couple of additional features of the framework.<\/p>\n\n\n\n
The first is a Delphi version agnostic mechanism for interrogating command line switches and obtaining values from them.  These are expected to be of the form:<\/p>\n\n\n\n
       -<<em>optionName<\/em>>[:<<em>value<\/em>>]<\/code> 
 or  -<<em>optionName<\/em>>[=<<em>value<\/em>>]<\/code> 
 or  -<<em>optionName<\/em>><\/code><\/p>\n\n\n\n
The second additional feature is ability to abort a test run.<\/p>\n\n\n\n
A third feature is the fact that an assert also returns its pass\/fail value back to the caller which may be used to direct conditional flow where appropriate.  In this case the result of a test is also used to determine whether or not to abort the remaining test run.<\/p>\n\n\n\n
An aborted test run still actually performs all remaining tests<\/em> but every single test that follows is recorded with an outcome of “skipped” rather than pass or fail.  Even raising an exception will not prevent subsequent tests from running, only those in the current test method.<\/p>\n\n\n\n
\n\n\n\n
I will be providing more documentation at some point as the project grows.  In the meantime, for a more detailed example you can peruse the self-test project<\/a> in the repo.  Be aware that this also takes advantage of some sneaky-stuff to access methods designed and intended for use specifically (and only) in self-test scenarios.<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span>  6<\/span> minutes]<\/span><\/span>Meet the new Smoketest.  Nothing like the old Smoketest.  A new, ultra-lightweight unit testing framework with xUnit 2.x result output (and the ability to support other output formats as needed).<\/p>\n","protected":false},"author":2,"featured_media":2957,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Introducing: Smoketest 2.0.  A new, ultra-lightweight unit testing framework for Delphi, with xUnit 2 formatted result output.","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[324,323,4,321,48],"tags":[292,51,295],"class_list":["post-2956","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-automation","category-ci-cd","category-delphi","category-devops","category-testing","tag-delphi","tag-smoketest","tag-testing"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/vape_2.jpg?fit=1229%2C314&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-LG","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2956","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=2956"}],"version-history":[{"count":10,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2956\/revisions"}],"predecessor-version":[{"id":2975,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2956\/revisions\/2975"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media\/2957"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=2956"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=2956"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=2956"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}