{"id":2208,"date":"2013-12-06T15:09:00","date_gmt":"2013-12-06T03:09:00","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=2208"},"modified":"2013-12-06T19:07:20","modified_gmt":"2013-12-06T07:07:20","slug":"old-xml-bug-in-delphi-causes-new-problems","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/2208\/","title":{"rendered":"Old XML Bug in Delphi Causes New Problems"},"content":{"rendered":"[Estimated Reading Time: <\/span> 4<\/span> minutes]<\/span><\/span>

A post came up in recent days on the NZ DUG mailing list<\/strong>, about a problem with the LoadXMLData()<\/strong> function on Android. The problem subsequently was found to also exist on Win32. And indeed, the cause was found to go back at least as far as Delphi 2006. So why did it only come up now ?<\/p>\n

<\/p>\n

The problem was identified as a result of someone trying to use LoadXMLData()<\/strong> using a string containing XML which contained an XML declaration which contained an atypical but perfectly valid XML encoding specification:<\/p>\n

\r\n  <?xml version='1.0' encoding = 'UTF-8' ?>\r\n<\/pre>\n
Do you see the problem ?<\/p>\n
It is the spaces either side of the ‘=’ in the encoding declaration.<\/p>\n
This causes problems when the XML loading code path for a string ends up calling CheckEncoding()<\/strong> (in the XMLDoc<\/strong> unit), which contains a shockingly naive attempt to remove the XML encoding declaration for anything other than a WideChar<\/strong> based encoding scheme:<\/p>\n
\r\nprocedure CheckEncoding(var XMLData: DOMString; const ValidEncodings: array of string);\r\nvar\r\n  Encoding: string;\r\n  EncodingPos, EncodingLen: Integer;\r\nbegin\r\n  { Check if the XML data has an encoding, if so it must match one of the\r\n    valid encodings, or we will remove it. }\r\n  Encoding := ExtractAttrValue(SEncoding, Copy(XMLData, 1, 50), '');\r\n  if (Encoding <> '') and not EncodingMatches(Encoding, ValidEncodings) then\r\n  begin\r\n    EncodingPos := Pos(SEncoding, XMLData);\r\n    EncodingLen := Length(Encoding) + 12;\r\n    Delete(XMLData, EncodingPos - 1, EncodingLen);\r\n  end;\r\nend;\r\n<\/pre>\n
The problem is that assumed length of the "encoding='..'"<\/code> declaration, allowing a fixed 12<\/strong> characters on top of the length of the encoding value itself.  Those 12 characters are intended to cover:<\/p>\n
\r\n               1    a leading space\r\n  encoding     8    for the word "encoding"\r\n  =            1    for the "=" symbol\r\n  ''           2    for the quotes around the encoding value\r\n<\/pre>\n
In the case of the specific example (which was a result from a call to a web service and so not directly under the control of the developer in question, as far as I know<\/em>), this breaks since it does not cleanly remove the encoding declaration and instead results in:<\/p>\n
\r\n  <?xml version='1.0'8' ?>\r\n<\/pre>\n
Which breaks the subsequent parsing of the XML for obvious reasons.<\/p>\n
Specification Interpretation<\/h2>\n
I suspect a mistaken interpretation of the XML specification<\/a> lies behind this long-standing mistake, since the specification of the XML declaration entity identifies the encoding declaration<\/a> as:<\/p>\n
\r\n[80]  EncodingDecl ::= S 'encoding' Eq ('"' EncName '"' | "'" EncName "'" )\r\n<\/pre>\n
Which on first blush would suggest that there can be no white-space either side of the “Eq<\/strong>“.  Except that if you follow the link to the Eq<\/strong> definition itself you find not a simple ‘=<\/strong>‘ symbol but:<\/p>\n
\r\n[25]   \tEq ::= S? '=' S?\r\n<\/pre>\n
i.e. an ‘=’ symbol with any amount of optional whitespace (not just spaces) either side.<\/p>\n
Mitigating Circumstances<\/h2>\n
As I say, this problem goes back at least as far as Delphi 2006, but up until recently it could be easily avoided.<\/p>\n
You see, when you load the XML into a TXMLDocument<\/strong>, there is not – as you might expect – a single, consistent code path followed for loading XML from the various supported sources (string variable, TStrings object and stream – i.e. from a file).<\/p>\n
Instead there are different code paths for each of these.<\/p>\n
In earlier versions of Delphi even using a simple string variable would invoke one of two<\/em> different code paths, depending on whether the string was an ANSIString<\/strong> or a WideString<\/strong> (DOMString).<\/p>\n
WideString<\/strong> XML is loaded via the simple string path and removing non-WideChar encodings from such strings makes sense in that context.<\/p>\n
ANSIString<\/strong> XML is loaded via the stream mechanism, bypassing the gotcha lurking in CheckEncoding()<\/strong>.<\/p>\n
LoadXMLData()<\/strong> is overloaded to support the two different string types explicitly, so as long as you keep an eye on what your string actually contains and declare the string type appropriately you could ensure that you invoked the ANSIString<\/strong> version of LoadXMLData()<\/strong> and you don’t have a problem.<\/p>\n
Alas Poor UTF-8.  I Knew Him Well<\/h2>\n
He hath borne me on his back a thousand times.<\/em><\/p>\n
In Delphi XE5<\/strong> however, Embarcadero<\/strong> chose to drop support for UTF-8 Strings on “NEXTGEN” platforms, and it is this decision I think which has created the problem here.  The ANSIString<\/strong> version of LoadXMLData()<\/strong> only really existed to support UTF-8 strings and so that overload is now subject to a conditional compilation directive.<\/p>\n
It is not supported by NEXTGEN<\/strong> compilers, including that for Android<\/strong>.<\/p>\n
The result is that even if you carefully declare an ANSIString<\/strong> to hold your XML, when you pass it to LoadXMLData()<\/strong> it will be converted to a WideString<\/strong> and go barelling off down the code path that leads to the flawed CheckEncoding()<\/strong>.<\/p>\n
You cannot even create a TXMLDocument<\/strong> and call LoadFromXML(ANSIString)<\/strong> directly since this too is not available to NEXTGEN<\/strong> compilers.<\/p>\n
You will of course get a warning about an implicit string conversion, but I cannot speak for the attitude to such warnings on behalf of the original developer who came up against this problem.  As I say, the problem only came to my attention and interest via a public mailing list.  For all I know, they simply ignore such warnings and\/or don’t understand the consequences.  I don’t know.<\/p>\n
So what can you do ?<\/p>\n
You could create the TXMLDocument<\/strong> and assign your string to the XML<\/strong> property (TStrings), using the Text<\/strong> property of course:<\/p>\n
\r\n  doc := TXMLDocument.Create(NIL);\r\n  doc.XML.Text := sMyXMLString;\r\n<\/pre>\n
You’ll still get a warning about the string conversion, but the loading of XML from TStrings<\/strong> bypasses CheckEncoding()<\/strong> so you won’t get the problem arising from the corruption of the XML declaration this can cause.<\/p>\n
However, TStrings<\/strong> is of course – these days – also WideString<\/strong> based so even if sMyXMLString<\/strong> is formally declared ANSIString<\/strong> (is this even allowed on Android?  I don’t know because I don’t use Delphi for FireMonkey for Android<\/em> myself) it will end up being converted to a UTF-16 string again.<\/p>\n
So although this bypasses the CheckEncoding()<\/strong> flaw, it will possibly cause subsequent problems as a result of the UTF-16 encoded XML incorrectly identifying itself as UTF-8 encoded.<\/p>\n
The Long Way Around<\/h2>\n
The developer with the problem identified that they could save the XML string to a file and then load it into a TXMLDocument from that file and this seemed to work, though whether the declared vs actual storage encoding issue  was either inadvertently or deliberately resolved or is simply not an issue in that case, I do not know.  Whatever the explanation, it’s certainly far from ideal (almost certainly unacceptable) to have to use an intermediate file to get a string in memory to an XML DOM in memory.<\/p>\n
Quite possibly the only reliable solution therefore is to implement your own mechanism for removing the XML encoding from the string (where appropriate to do so) and making sure you do that before<\/em> passing it to LoadXMLData()<\/strong>.<\/p>\n
I leave that as an exercise for the reader.  \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span>  4<\/span> minutes]<\/span><\/span>A post came up in recent days on the NZ DUG mailing list, about a problem with the LoadXMLData() function on Android. The problem subsequently was found to also exist on Win32. And indeed, the cause was found to go back at least as far as Delphi 2006. So why did it only come up […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[4,207],"tags":[292,265,264],"class_list":["post-2208","post","type-post","status-publish","format-standard","hentry","category-delphi","category-xe5","tag-delphi","tag-nextgen","tag-xml"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-zC","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2208","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=2208"}],"version-history":[{"count":7,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2208\/revisions"}],"predecessor-version":[{"id":2215,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2208\/revisions\/2215"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=2208"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=2208"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=2208"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}