{"id":2164,"date":"2013-11-19T17:43:53","date_gmt":"2013-11-19T05:43:53","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=2164"},"modified":"2013-11-19T17:43:53","modified_gmt":"2013-11-19T05:43:53","slug":"smoketest-some-differences-with-dunit","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/2164\/","title":{"rendered":"Smoketest – Some Differences With DUnit"},"content":{"rendered":"[Estimated Reading Time: <\/span> 6<\/span> minutes]<\/span><\/span>

Writing tests in Smoketest<\/strong> is intended to enable a test developer to write tests in a way that describe themselves, without requiring the test developer to add this “narrative” themselves. To see this in action, I thought I would compare some simple DUnit<\/strong> tests with the equivalent using the Smoketest<\/strong> framework.<\/p>\n

<\/p>\n

For this exercise we shall consider the test for a for splitting a string based on some delimiting character. The prototype for the function is:<\/p>\n

\r\n  class function WIDE.Split(const aString: UnicodeString;\r\n                            const aChar: WideChar;\r\n                            var aParts: TWideStringArray): Boolean;\r\n<\/pre>\n
In any successful call to this function there are a number of things that need to be checked to ensure correct behaviour.<\/p>\n
First, the function should return TRUE<\/strong> only if one or more instances of aChar<\/strong> are found in the string.  Second, where TRUE<\/strong> has been returned, the number of entries in the aParts<\/strong> array needs to be correct.  And finally, each item in that aParts<\/strong> array needs to be what we would expect.<\/p>\n
A DUnit<\/strong> test for this might look something like this:<\/p>\n
\r\nvar\r\n  ReturnValue: Boolean;\r\n  aString: UnicodeString;\r\n  aChar: WideChar;\r\n  aParts: TWideStringArray;\r\nbegin\r\n  aString := 'left*mid-left*middle*mid-right*right';\r\n  aChar   := '*';\r\n  ReturnValue := WIDE.Split(aString, aChar, aParts);\r\n\r\n  CheckEquals(ReturnValue, TRUE);\r\n  CheckEquals(Length(aParts), 5);\r\n  CheckEquals(aParts[0], 'left');\r\n  CheckEquals(aParts[1], 'mid-left');\r\n  CheckEquals(aParts[2], 'middle');\r\n  CheckEquals(aParts[3], 'mid-right');\r\n  CheckEquals(aParts[4], 'right');\r\nend;\r\n<\/pre>\n
First off all, it’s worth mentioning here that the IDE support for DUnit<\/strong> was actually counter-productive in this case.  The wizard fails to recognize a class function<\/strong> and creates swathes of boiler-plate test code for setting up and tearing down tests, instantiating the class in order to (incorrectly) call it via an instance.<\/p>\n
But apart from that, once all extraneous and erroneous code has been cleaned out, we can get on with writing the test itself and as written above – this all looks fine, right ?<\/p>\n
Wrong.<\/p>\n
What’s worse is that as long as all the tests pass there is no reason to suspect that this test is actually completely and utterly wrong<\/em>.  The CheckEquals<\/strong> method places particular significance on the order of the parameters identifying the two values that are supposed to be equal.  Without inspecting the parameter list for the CheckEquals()<\/strong> method this significance is not immediately obvious.<\/p>\n
When the two parameters are equal (test will pass) this doesn’t matter, but in the event of a test failure<\/em> if you get the order wrong then the test report will itself be misleading, reporting the actual value as the expected value and vice versa.  The correct DUnit<\/strong> test should be written:<\/p>\n
\r\nvar\r\n  ReturnValue: Boolean;\r\n  aString: UnicodeString;\r\n  aChar: WideChar;\r\n  aParts: TWideStringArray;\r\nbegin\r\n  aString := 'left*mid-left*middle*mid-right*right';\r\n  aChar   := '*';\r\n  ReturnValue := WIDE.Split(aString, aChar, aParts);\r\n\r\n  CheckEquals(TRUE, ReturnValue);\r\n  CheckEquals(5, Length(aParts));\r\n  CheckEquals('left', aParts[0]);\r\n  CheckEquals('mid-left', aParts[1]);\r\n  CheckEquals('middle', aParts[2]);\r\n  CheckEquals('mid-right', aParts[3]);\r\n  CheckEquals('right', aParts[4]);\r\nend;\r\n<\/pre>\n
And still, the only information coming from DUnit<\/strong> about these tests will be that an actual value either did or did not equal it’s expected value.  If the test developer wishes to add some descriptive information about this they must add it as a msg<\/strong> parameter to the test:<\/p>\n
\r\n  CheckEquals(TRUE, ReturnValue, 'Split()');\r\n  CheckEquals(5, Length(aParts), 'No. of parts');\r\n  CheckEquals('left', aParts[0], 'aParts[0]');\r\n  CheckEquals('left', aParts[1], 'aParts[1]');\r\n  etc\r\n<\/pre>\n
This always felt backwards to me with DUnit<\/strong> and was one of the primary reasons for creating Smoketest<\/strong> and taking an entirely different approach which more closely resembles the language we would use when describing our expected test outcomes.<\/p>\n
Let’s look at the equivalent tests in Smoketest<\/strong>:<\/p>\n
\r\n  Test('Split()').Expect(ReturnValue).IsTRUE;\r\n  Test.Expect(Length(aParts)).Equals(5);\r\n  Test('aParts')[0].Expect(aParts[0]).Equals('left');\r\n  Test('aParts')[1].Expect(aParts[1]).Equals('mid-left');\r\n  Test('aParts')[2].Expect(aParts[2]).Equals('middle');\r\n  Test('aParts')[3].Expect(aParts[3]).Equals('mid-right');\r\n  Test('aParts')[4].Expect(aParts[4]).Equals('right');\r\n<\/pre>\n
This is perhaps a little more verbose but to my mind reads far more naturally as an expression of our test expectations.  We say what<\/strong> it is we are testing (if it needs spelling out), identify the where<\/strong> the value comes from that we need to test and then say how<\/strong> it should meet our expectations.<\/p>\n
NOTE: <\/strong>The use of indexing syntax on the Test()[]<\/strong> expression is optional but facilitates labelling tests where tests are being applied iteratively from some collection of test vectors.  For example if we had declared our expected resulting parts in a VECTORS<\/strong> array:<\/p>\n
\r\n  const\r\n    VECTORS: array[0..4] of String = ('left', 'mid-left', 'middle', 'mid-right', 'right');\r\n\r\n  ..\r\n\r\n    for i := 0 to High(VECTORS) do\r\n      Test('aParts')[i].Expect(aParts[i]).Equals(VECTORS[i]);\r\n<\/pre>\n
With the context provided by the interfaces returned at each step along the way, the test developer is guided toward writing tests that are appropriate to the values being tested.<\/p>\n
But there is another advantage to the way that tests work in Smoketest<\/strong> as compared with DUnit<\/strong>.<\/p>\n
Fail Early.  But Not Too<\/strong> Early<\/h2>\n
In DUnit<\/strong>, each CheckEquals()<\/strong> must pass if the following checks are to be performed.  Sometimes this is desirable.  If you are testing that you have an object reference before you then go on to check other properties of that object then there is little point in proceeding since those tests are simply going to fail.<\/p>\n
This might be described as “Failing Early<\/em>“.<\/p>\n
But in other cases, the manner in which subsequent tests fail could provide useful diagnostic information to explain the initial failure.<\/p>\n
Consider a hypothetical situation where a developer has identified a potential optimisation in the Split()<\/strong> function.  They make their change and run the tests, but as a result of their change the Split()<\/strong> function creates the wrong number of items in the aParts<\/strong> array.<\/p>\n
As a result in DUnit<\/strong>, this part of the test will fail and case the test method to halt:<\/p>\n
\r\n  CheckEquals(5, Length(aParts));\r\n<\/pre>\n
If – say – the Split()<\/strong> function is creating only 4 items in the aParts<\/strong> array, then what those 4 parts contain could contain useful information that will help the developer realise their mistake.  With DUnit<\/strong> they won’t get this information.<\/p>\n
With Smoketest<\/strong> – by default – the test of the number of items in aParts<\/strong> will fail but the test will continue to apply the further tests and will either output garbage, crash or halt with an ERangeCheck<\/strong> exception (if compiling the tests with range checking enabled) only on the test of the fifth, non-existent item in the aParts<\/strong> array.  As a result we might see the following in our test results:<\/p>\n
\r\n  No. of parts - FAILED\r\n    Expected: 5\r\n    Actual: 4\r\n\r\n  aParts[0] - FAILED\r\n    Expected: 'left'\r\n    Actual: 'left*m';\r\n\r\n  aParts[1] - FAILED\r\n    Expected: 'mid-left'\r\n    Actual: 'id-left*mi';\r\n\r\n  aParts[2] - FAILED\r\n    Expected: 'middle'\r\n    Actual: 'ddle*mid';\r\n\r\n  aParts[3] - FAILED\r\n    Expected: 'mid-right'\r\n    Actual: 'dle*mid-';\r\n\r\n  ERANGECHECK EXCEPTION\r\n<\/pre>\n
This is not the actual output, just a representation of it.  And it is entirely hypothetical data of course not intended to indicate any particular type of error that might exist in a function such as Split()<\/strong> rather only to demonstrate that “fail early<\/em>” is not always the most helpful strategy, especially when testing.<\/p>\n
We should always of course “Fix the first problem<\/em>“, but sometimes the consequential problems help us identify what that first problem is<\/em>.<\/p>\n
With Smoketest<\/strong> you can get DUNit<\/strong>-like fail early behaviour if you want it.  And more.<\/p>\n
To make this test halt if the number of items in the aParts<\/strong> array is not what we expect, then we simply add a qualification to the effect that this test result is a required outcome and we add it to the test itself:<\/p>\n
\r\n  Test.Expect(Length(aParts)).Equals(5).IsRequired;\r\n<\/pre>\n
As explained in an earlier post, an IsRequired<\/strong> result will halt the current test method<\/strong> if the test fails.  IsCritical<\/strong> can be used to halt an entire test case<\/strong>, and IsShowStopper will halt the entire test run<\/em><\/strong>.<\/p>\n
But so far all we have really seen is how Smoketest<\/strong> does what DUnit<\/strong> also does, just differently.  Now for something completely different.<\/p>\n
Greater Expressiveness<\/h2>\n
With DUnit<\/strong> the number and type of tests you can perform is fairly limited.  The CheckEquals()<\/strong> method is called upon to carry a great deal of the burden of testing, often hiding the detail of a test in the expression used to calculate a result passed to that CheckEquals()<\/strong> method as a boolean.<\/p>\n
Imagine a scenario where a test was interested only in whether or not some value exceeded some threshold amount but was not concerned with the precise value.  In other words, that some value was greater than some other value.<\/p>\n
In DUnit<\/strong> you would write this test as follows:<\/p>\n
\r\n  CheckEquals(TRUE, value > limit);\r\n<\/pre>\n
And in the event of a test failure you get the not very helpful report that TRUE<\/strong> was expected, not FALSE<\/strong>.  So you are forced to add some narrative to describe the test to endow it with a meaning that is no apparent from the test itself:<\/p>\n
\r\n  CheckEquals(TRUE, value > limit, 'Value is greater than Threshold);\r\n<\/pre>\n
In Smoketest<\/strong>, because test expectations are specific and appropriate to the type of value being tested, there is far greater diversity and richness of expression in the tests available, enabling tests to be written in a way that describe themselves.  This limit test for example would be written in Smoketest<\/strong> using an Integer<\/strong> expectation:<\/p>\n
\r\n  Test('value').Expect(value).IsGreaterThan(limit);\r\n<\/pre>\n
Not only does Smoketest<\/strong> guide us toward writing a more appropriate test but since the test describes itself the result is now actually more compact than DUnit<\/strong> where the test has to be described separately from and in addition to the test itself (and then only if the test developer could be bothered to add that description in the first place).<\/p>\n
As I mentioned, in this particular case the DUnit<\/strong> IDE wizard created a whole lot of boilerplate code for setting up and tearing down this test case that was wholly inappropriate on this occasion.  But there are times when you need such housekeeping and this is what I shall cover in my next Smoketest<\/strong> post.<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span> 6<\/span> minutes]<\/span><\/span> Writing tests in Smoketest is intended to enable a test developer to write tests in a way that describe themselves, without requiring the test developer to add this “narrative” themselves. To see this in action, I thought I would compare some simple DUnit tests with the equivalent using the Smoketest framework.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":[]},"categories":[4],"tags":[292,51,295],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-yU","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":2193,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2193\/","url_meta":{"origin":2164,"position":0},"title":"DUnit Compatibility in Smoketest","date":"25 Nov 2013","format":false,"excerpt":"To address some odd concerns about differences between DUnit and Smoketest, I thought it would be useful to demonstrate how it is entirely within the gift of a Smoketest user to create their own \"comfort\" layer, to make using Smoketest more similar to the DUnit framework if they wish (though\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2169,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2169\/","url_meta":{"origin":2164,"position":1},"title":"Smoketest – Set Me Up \/ Tear Me Down","date":"20 Nov 2013","format":false,"excerpt":"In a previous post I demonstrated how the default \"pretty name\" for a Smoketest test case (derived from the test case classname) can be over-ridden by a test developer by implementing a specific interface (INameCase) on the test case class itself. There are some other interfaces that can be implemented\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2095,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2095\/","url_meta":{"origin":2164,"position":2},"title":"Extending Smoketest (Part 1) – An Inspector Calls","date":"05 Nov 2013","format":false,"excerpt":"In the soon to be released Smoketest framework it is sometimes useful to create new test types to supplement the tests built-in to the framework. In this and the next post I will walk through the process of implementing and registering a custom test with the Smoketest framework. In a\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2006,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2006\/","url_meta":{"origin":2164,"position":3},"title":"Blowing Smoke…","date":"01 Nov 2013","format":false,"excerpt":"To alleviate the grind of polishing and sanitising my code (and, let's be honest, just plain 'fixing' it in some cases) ready for release, I have re-kindled my participation on Stack Overflow. In a happy confluence yesterday a question came up which allowed me to exercise one of the libraries\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"Ready to Run","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/Screen-Shot-2013-11-01-at-09.00.33.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":1233,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/1233\/","url_meta":{"origin":2164,"position":4},"title":"ReverseBytes()","date":"20 Sep 2012","format":false,"excerpt":"In some comments on StackOverflow, Jeroen asked me to post my code for reversing bytes. Rather than posting code into that question\/answer that wasn't directly relevant to the question\/answer, I decided to quickly throw the code up on here. The intent with ReverseBytes() is - as the name says -\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2111,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2111\/","url_meta":{"origin":2164,"position":5},"title":"Extending Smoketest (Part 2) – Great(er) Expectations","date":"07 Nov 2013","format":false,"excerpt":"In my previous post on Smoketest I showed how you can extend the inspections framework to work with complex types in your code. As promised, I shall now show how you can do much the same thing to extend the framework with entirely new tests. \u201cI must be taken as\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2164"}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=2164"}],"version-history":[{"count":4,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2164\/revisions"}],"predecessor-version":[{"id":2168,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2164\/revisions\/2168"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=2164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=2164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=2164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}