{"id":563,"date":"2009-09-26T23:55:32","date_gmt":"2009-09-26T11:55:32","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=563"},"modified":"2009-09-26T23:55:32","modified_gmt":"2009-09-26T11:55:32","slug":"delphi-unicode-wide-ansi","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/563\/","title":{"rendered":"Delphi Unicode = Wide-ANSI"},"content":{"rendered":"[Estimated Reading Time: <\/span> 7<\/span> minutes]<\/span><\/span>

Be careful what you wish for. A lot of people were overjoyed to hear that Unicode support was coming to Delphi. Some were skeptical of the chosen implementation approach however, it all seemed just a little bit too<\/em> easy. I was one, and sadly it seems I was right.
\n
\nI’ve just started updating a whole host of code to Delphi 2009. \u00a0Since Unicode is what my code has to speak whether I like it or not (if I want to use the latest\/greatest Delphi compilers) then I may as well get with the program and drag my ANSI code kicking and screaming into the UTF-16 world.<\/p>\n

“Kicking and Screaming” is certainly involved. \u00a0But mostly it’s me doing it.<\/p>\n

In utter frustration.<\/p>\n

There are currently two head-shaped dents in the wall. Allow me to share my pain.<\/p>\n

Dent #1: UTF8ToANSI()<\/h2>\n

The help says:<\/p>\n

Call Utf8ToAnsi to convert a UTF-8 string to Ansi. S is a string encoded in UTF-8. Utf8ToAnsi returns the corresponding string that uses the Ansi character set.<\/p><\/blockquote>\n

Well if you do<\/span> call UTF8ToAnsi <\/strong>and expect it to do what that says it will do then you are in for a disappointment. \u00a0Because it actually returns a UnicodeString<\/strong>.<\/p>\n

Not “what it says on the tin” and sure as eggs is eggs not what the caller wants or expects, not matter how quickly and dirtily they want to hack their old ANSI code into a warning-free ANSI-on-WideAPI “pseudo-Unicode” application..<\/p>\n

Dumb. \u00a0Dumb. \u00a0Dumb.<\/p>\n

But it gets even dumber. There is<\/em> now also a function UTF8ToString()<\/strong>! And if we inspect the source for that we find the following puzzling implementation:<\/p>\n

\r\n  function UTF8ToString(const S: RawByteString): string;\r\n  begin\r\n  {$IFDEF UNICODE}\r\n    Result := UTF8ToUnicodeString(S);\r\n  {$ELSE}\r\n    Result := UTf8ToAnsi(S);\r\n  {$ENDIF}\r\n  end;\r\n<\/pre>\n
You may be thinking that this isn’t puzzling at all.  That it is a perfectly sensible implementation – if we’re compiling with UNICODE defined then defer to the UnicodeString implementation.  Otherwise defer to the ANSI implementation.<\/p>\n
But hang-on, since when was UNICODE an option in Delphi 2009?  CodeGear decided that it was to be “Unicode or the Hi-rode”.<\/p>\n
But it get’s better, because of course in deferring to the ANSI implementation – even assuming this is ever compiled with UNICODE not<\/strong> defined – they are of course calling the so-called-ANSI implementation that – UNICODE or not – calls the Unicode implementation!<\/p>\n
You can’t help but think that CodeGear managed to confuse even themselves with their approach to Unicode.<\/p>\n
OK, But What’s this Wide-ANSI Nonesense ?<\/h2>\n
Ok, so UTF8ToANSI()<\/strong> is not something that I’m guessing many people are actually using (my code is working with the Apple Bonjour SDK<\/a> which makes extensive use of UTF-8, and the code originated in Delphi 7, hence the ANSI\/UTF8 conversions).<\/p>\n
So I ran into an edge-case<\/a>.  No big deal, right?  Granted.  But sadly there are other, wider [sic] issues which people will<\/em> undoubtedly run into.<\/p>\n
Let me give you an example that I’m certain will be encountered by more people than that UTF8\/ANSI scenario.<\/p>\n
Dent #2: Uppercase()<\/h2>\n
\r\nvar ws: String;  \/\/ UnicodeString in Delphi 2009\/10\r\n\r\nws := 'a\u00e0';\r\nws := Uppercase(ws);\r\n<\/pre>\n
What does ws<\/strong> contain after the call to Uppercase()<\/strong>? \u00a0If you said\u00a0A\u00c0<\/strong> then congratulations – you are 100% wrong<\/span>. \u00a0It actually contains\u00a0A\u00e0<\/strong>.<\/p>\n
Sadly, that is NOT the uppercase version of the input string according to the Unicode specification.<\/p>\n
Now, to be fair, the documentation in this case is quite correct and clear that Uppercase() does not handle Unicode Strings as Unicode.  Which makes an absolutely mockery of a strongly typed language such as Delphi.<\/p>\n
Uppercase()<\/strong> accepts a String<\/strong> parameter.  In Delphi 2009 String<\/strong> is a Unicode data type.<\/p>\n
In Delphi 2009+ Uppercase()<\/strong> is – by contract – a Unicode function and should jolly well behave as such.<\/p>\n
Frankly, this decision by CodeGear speaks to me of a cavalier disregard for what makes Delphi, well, Delphi.<\/p>\n
Anyway… Now, the hard part.  Can you guess what function you should<\/em> use to properly convert a UnicodeString to uppercase?<\/p>\n
Well, you actually have a choice of two:<\/p>\n
\r\n    WideUppercase( );\r\n    ANSIUppercase( );\r\n<\/pre>\n
No, wait.  That can’t be right surely?  ANSI<\/strong>Uppercase?<\/p>\n
Yes.  Of course the alarm bells should be ringing as soon as you realise that ANSIUppercase<\/strong> accepts not an ANSIString<\/strong> parameter but a plain String<\/strong> parameter, which of course made perfect sense pre-Delphi 2009 when String<\/strong> meant ANSIString<\/strong>, but in Delphi 2009+ CodeGear found themselves painted into a corner.<\/p>\n
Having decided that they, and only they, would get to choose what String<\/strong> meant to the compiler (although I think that the $ifdef UNICODE in System.pas is evidence that they weren’t – at one time at least – as convinced that this was the right and obvious thing to do as they seemed to suggest), if they properly modified these ANSI RTL functions to accept ANSI strings, then a lot of code currently calling them with String<\/strong> parameters would start throwing up warnings when String<\/strong> transformed from ANSIString<\/strong> to UnicodeString<\/strong>.<\/p>\n
But at the same time the vast, and I mean VAST majority of code would simply be calling Uppercase()<\/strong>, and they didn’t want any warnings coming from that either.<\/p>\n
What a pickle.<\/p>\n
Fight Fire With Fire<\/h2>\n
It seems they chose to resolve this dilemma by adding a little more confusion into the mix, creating a situation where someone explicitly calling an ANSI routine will obtain a Wide operation and someone calling the “default” implementation, presumably expecting it to yield a Wide operation (because Delphi 2009 is Unicode through and through, right?) is rewarded with an ANSI operation.<\/p>\n
Perhaps they thought that if they made our heads spin enough we wouldn’t notice what a pigs-ear they’d made of the whole thing?<\/p>\n
But what does all this mean for someone with some old apps that they can’t wait to get updated to the latest Unicode compiler so that they can start selling to their customers who have been demanding Unicode support?<\/p>\n
Well having compiled your pre-Unicode source with the Unicode compiler you may have been very happy to find that you had only a handful of warnings to deal with and then BINGO, you had a Unicode application.<\/p>\n
Sorry.<\/p>\n
I hate to break it to you, but you may not have finished yet.<\/p>\n
What you have at the moment is an application that in all likelihood is still behaving very much like an ANSI application, it’s just that it’s now sitting atop the Windows Wide API’s, pretending<\/em> to be a Unicode application.<\/p>\n
In many respects it may fool you, perhaps for a long time, because anyone who hadn’t previously tackled Unicode head-on almost certainly doesn’t have any actual need for Unicode and their application will not be stressing those parts of Unicode support that actually separate a “Unicode” application from a “non-Unicode” application.<\/p>\n
A Thought Experiment<\/h2>\n
This is a “Thought Experiment” in the sense of “You know what Thought did?”  One answer to which is “He thought he did, but he didn’t.”<\/p>\n
Let’s take a simple and I think highly common situation.  A text entry field for some user specified code that is required to accept only alpha-numeric characters and enforce uppercase.<\/p>\n
Pre-Delphi 2009 such a field may well have had a key filter installed in an event handler:<\/p>\n
\r\n  procedure TForm1.Edit1KeyPress(Sender: TObject; var Key: Char);\r\n  begin\r\n    if (Key in ['a'..'z']) then\r\n      Key := UpCase(Key)\r\n    else if NOT (Key in ['0'..'9']) then\r\n      Key := #0;\r\n  end;\r\n<\/pre>\n
In Delphi 2009 this throws up the “WideChar reduced to byte char” warning that everyone “doing Unicode” in Delphi has been talking about<\/a>, and like a good little Delphi developer they do what CodeGear tell them and change this code.  But let us imagine that we know a little bit about Unicode and are actually migrating to Delphi 2009 because we want to be able to market our product as a Unicode application.<\/p>\n
So rather than simply calling CharInSet<\/strong>, because that still only deals with ASCII characters, we adjust the routine to call the Windows Unicode support routine IsCharAlphaNumeric<\/strong> instead:<\/p>\n
\r\n  procedure TForm1.Edit1KeyPress(Sender: TObject; var Key: Char);\r\n  begin\r\n    if IsCharAlphaNumeric(Key) then\r\n      Key := UpCase(Key)\r\n    else\r\n      Key := #0;\r\n  end;\r\n<\/pre>\n
With this simple change the code now compiles without any warnings.<\/p>\n
YAY!  We didn’t fall into the trap laid for us by CharInSet!  We dealt with this properly and now we have a Unicode application!<\/p>\n
Don’t we?<\/p>\n
No.<\/p>\n
Can you guess what that WideChar<\/strong> version of UpCase()<\/strong> does.  Yep.  Behaves exactly the same as the ANSI version.<\/p>\n
Dumb. Dumb. Dumb.<\/p>\n
There is not even the excuse of needing to maintain backward compatability in this case – there was no WideChar version of UpCase()<\/strong> prior to Delphi 2009.<\/p>\n
Now, anyone who understands Unicode and specifically the properties and characteristics of UTF16 will be able to tell you why UpCase( WideChar )<\/strong> does not perform a Unicode case conversion – it simply can<\/strong>not.  A single WideChar may not represent an entire character – it may be part of a surrogate pair and whilst I don’t think that there are any case convertible characters that require surrogate pairs currently, that could change (and I may be wrong on that anyhow).<\/p>\n
So what else could it do but echo the ANSI implementation?<\/p>\n
Well one option would have been to reflect the true nature of Unicode and simply not try to create the illusion of supporting something that cannot be supported.<\/p>\n
But more acceptable I think would have been to perform a Unicode conversion on those chars that it could (non-surrogates) and if a surrogate was supplied as input, simply return it unmodified.<\/p>\n
As it is, the previous (and current) “ANSI” implementation is incomplete w.r.t ANSI, providing only ASCII case conversion.  It would have been easy to argue that a Unicode implementation that only operated on characters in the BMP (Basic Multilingual Plane) was the natural and obvious behaviour for a Wide UpCase implementation.<\/p>\n
So I’m afraid if you do want and\/or need proper Unicode support, you’ve still got some work to do before you can get there, and unfortunately the compiler is not going to help you from this point on.  Furthermore, the VCL now seems to go out of it’s way to make it harder by in some cases completely breaking the type-safety that you can normally expect when working in Delphi and which would have guided you toward the answers to the numerous questions that arise when contemplating proper Unicode support.<\/p>\n
The compiler assistance in “helping you find the things you need to change for Unicode support” only really works if you don’t actually need proper<\/em> Unicode support and just want to get your ANSI code running over the Wide API in Windows.<\/p>\n
Once you’ve reached that point – or even before that – if you then decide you want to do Unicode properly, I fear you will find that the design decisions made to facilitate the migration of Wide-ANSI applications will frustrate you and complicate your job no-end.<\/p>\n
Really, I have to wonder if it really was worth getting so excited about Unicode support if it’s main audience is people not actually supporting Unicode properly?<\/p>\n
I can only hope that the 64-bit support that it seems people increasingly need is not being delayed in order to make way for a cross-platform implementation that will suffer the same identity crisis as the Unicode implementation is littered with.<\/p>\n
Getting it done quick sometimes is not as important as doing it right.<\/p>\n
In the case of Unicode I’m afraid I’ve not come across anything to make me think it was “done right” at all.<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span> 7<\/span> minutes]<\/span><\/span> Be careful what you wish for. A lot of people were overjoyed to hear that Unicode support was coming to Delphi. Some were skeptical of the chosen implementation approach however, it all seemed just a little bit too easy. I was one, and sadly it seems I was right.<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":[]},"categories":[4],"tags":[292,22],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-95","jetpack_sharing_enabled":true,"jetpack-related-posts":[{"id":426,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/426\/","url_meta":{"origin":563,"position":0},"title":"The “Big Switch”","date":"28 Apr 2009","format":false,"excerpt":"Much has been made in the past and again more recently, about the lack of a compiler switch in Delph 2009 to govern the behaviour of the String type. CodeGear have repeatedly said that it was not possible\/practical to provide such a switch, but their advice to anyone concerned about\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":375,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/375\/","url_meta":{"origin":563,"position":1},"title":"Delphi 2009 – StringPerformance Redux","date":"22 Sep 2008","format":false,"excerpt":"It looks like I may have jumped the gun with my conclusions from the previous exercise to benchmark string performance in Delphi 2009.\u00a0 Following a useful exchange in the comments with Kryvich I corrected a small discrepancy in the tests and made some changes to the performance testing subsystem within\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/delphi2009-stringperformance-chart.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":349,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/349\/","url_meta":{"origin":563,"position":2},"title":"Delphi 2009 – String Performance","date":"18 Sep 2008","format":false,"excerpt":"NOTE: Downloads are now fixed! Andreas Hausladen generously took the time to make some detailed comments on my previous post, one of which prompted me to throw together some further performance test cases for String types specifically.\u00a0 The results were something of a mixed bag and contained some surprises. The\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/delphi2009-stringperformance-resultscapture.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":338,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/338\/","url_meta":{"origin":563,"position":3},"title":"Delphi 2009 – A Heads-Up for Low-Level Coders","date":"13 Sep 2008","format":false,"excerpt":"Prompted by a conversation with some colleagues where-in we collectively speculated about the implementation details of a generic class and what impact - if any - this might have on performance vs a \"traditional\" polymorphic equivalent, I threw together a quick performance test case in my Smoketest framework, and as\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2006,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2006\/","url_meta":{"origin":563,"position":4},"title":"Blowing Smoke…","date":"01 Nov 2013","format":false,"excerpt":"To alleviate the grind of polishing and sanitising my code (and, let's be honest, just plain 'fixing' it in some cases) ready for release, I have re-kindled my participation on Stack Overflow. In a happy confluence yesterday a question came up which allowed me to exercise one of the libraries\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"Ready to Run","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/Screen-Shot-2013-11-01-at-09.00.33.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":2185,"url":"https:\/\/www.deltics.co.nz\/blog\/posts\/2185\/","url_meta":{"origin":563,"position":5},"title":"Smoketest – Performance Case Visualisations","date":"22 Nov 2013","format":false,"excerpt":"This post is a peek behind the curtain of the next major update to Smoketest which I hope to have completed shortly: Performance Case visualisations. Smoketest has always had two types of test case that you could implement by deriving from two distinct base classes: TTestCase is the base class\u2026","rel":"","context":"In "Delphi"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/Screen-Shot-2013-11-22-at-09.57.52.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/563"}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=563"}],"version-history":[{"count":10,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/563\/revisions"}],"predecessor-version":[{"id":667,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/563\/revisions\/667"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=563"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=563"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=563"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}