{"id":2936,"date":"2019-09-19T10:30:10","date_gmt":"2019-09-18T22:30:10","guid":{"rendered":"https:\/\/www.deltics.co.nz\/blog\/?p=2936"},"modified":"2019-09-19T10:38:05","modified_gmt":"2019-09-18T22:38:05","slug":"azure-devops-iterative-insertion-fixed","status":"publish","type":"post","link":"https:\/\/www.deltics.co.nz\/blog\/posts\/2936\/","title":{"rendered":"Azure DevOps – Iterative Insertion Fixed!"},"content":{"rendered":"[Estimated Reading Time: <\/span> 4<\/span> minutes]<\/span><\/span>\n

In yesterday’s post<\/a> I mentioned that I had been unsuccessful in using Iterative Insertion<\/strong><\/a> to simplify building for multiple Delphi<\/strong> versions. Well, as so often happens, I had a flash of insight overnight and found the solution this morning!<\/p>\n\n\n\n\n\n\n\n

In the foot note of that previous post, I mentioned having used a different trick to preserve per-Delphi version specific build steps in my build jobs (to ease the process of identifying build failures in that eventuality).<\/p>\n\n\n\n

The ‘trick’ was to create a local template for my builds. This template is specific to my project build needs so doesn’t form part of the azure-pipeline-templates<\/a> library but sits in my repo alongside my main build pipeline:<\/p>\n\n\n\n

parameters:\n  delphiVersion: ''\n  platform: 'x86'\n\nsteps:\n- template: delphi-build.yml@templates\n  parameters:\n    delphiVersion: ${{ parameters.delphiVersion }}\n    platform: ${{ parameters.platform }}\n    project: tests\\Tests\n    preBuildInline: duget restore --noUnpack\n    postBuildInline: .bin\\Tests -f=.results\\Delphi.${{ parameters.delphiVersion }}.xml\n<\/code><\/pre>\n\n\n\n
This template in turn references the main delphi-build.yml<\/a><\/strong> template in GitHub.  It provides most of the parameters I need that do not vary regardless of the different Delphi versions I am building for.  It achieves variability in parameter values where needed by using template substitution.<\/p>\n\n\n\n
This is most obvious in the delphiVersion<\/code><\/strong> parameter, which is passed through from this template to the delphi-build<\/strong> template.  But it is also used to ensure that test results produced after a successful build are named for that delphiVersion<\/strong> also.<\/p>\n\n\n\n
This simple template already would allow me to greatly simplify my main build.yml<\/strong> for this project:<\/p>\n\n\n\n
trigger:\n- develop\n- master\n\nresources:\n  repositories:\n    - repository: templates\n      type: github\n      name: deltics\/azure-pipeline-templates\n      ref: refs\/heads\/develop\n      endpoint: GitHubTemplates\n\npool:\n  name: 'The Den'\n\nsteps:\n- template: gitversion.yml@templates\n\n- script: |\n    duget init\n    duget restore --path tests\n  displayName: 'Initialise DuGet'\n\n# x86 builds ---------------------------------\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: 7\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: 2010\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: xe4\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: 10.3\n\n# x64 builds ---------------------------------\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: xe4\n    platform: x64\n\n- template: build-with-version.yml\n  parameters:\n    delphiVersion: 10.3\n    platform: x64<\/code><\/pre>\n\n\n\n
Still pretty verbose, but a vast improvement on the original and much easier to maintain.<\/p>\n\n\n\n
You may note that my reference to the azure-pipeline-templates<\/strong> GitHub repository in this pipeline includes a ref<\/code> property which results in this pipeline using the latest version from the develop<\/strong> branch, as I am developing this pipeline alongside making refinements to that GitHub template.  Once finalised (i.e. develop<\/strong> merged to master<\/strong>), this ref<\/code> will be removed from this pipeline.<\/p>\n\n\n\n
x86<\/strong> and x64<\/strong> builds still have to be specified separately as there is no assumption by the downstream templates that I wish to build for both architectures even if the specified Delphi compiler version supports x64<\/strong>.<\/p>\n\n\n\n
The next step is to introduce a further template which can ‘stamp out’ these individual build steps for me.<\/p>\n\n\n\n
It turns out that the required template is itself very simply.  I had made two mistakes yesterday when first attempting this.  The first was not double-checking my (still developing) knowledge of YAML and ensuring that my array syntax was correct.  The second was over-looking a trivial but significant element in my iterative insertion syntax.<\/p>\n\n\n\n
In my defence, the errors these mistakes resulted in (e.g. mapping not allowed here<\/em>) were again not especially helpful.<\/p>\n\n\n\n
Here’s the template I ended up with:<\/p>\n\n\n\n
parameters:\n  delphiVersions: []\n  platform: 'x86'\n\nsteps:\n- ${{ each version in parameters.delphiVersions }}:\n  - template: build-with-version.yml\n    parameters:\n      delphiVersion: ${{ version }}\n      platform: ${{ parameters.platform }}<\/code><\/pre>\n\n\n\n
This template assumes that I am building for x86<\/strong> but allows me to specify some other platform<\/code> in a parameter.  It also expects that I will provide an array<\/em> of delphiVersions<\/code> that I wish to build.<\/p>\n\n\n\n
It then uses the iterative insertion feature of Azure DevOps pipelines to add a step for each of those delphiVersions<\/code>.  Each of these steps uses the build-with-version.yml<\/strong> local template.<\/p>\n\n\n\n
I could have ‘bypassed’ the build-with-version<\/strong> template and instead directly referenced the delphi-build.yml<\/strong> template, but this would only make sense if I were to then git rid of the build-with-version.yml<\/strong> entirely, otherwise I would have to keep both templates in sync if I ever needed to change the build properties for this project.  This way, I know that build-with-version.yml<\/strong> is the one place where my project build is essentially configured.<\/p>\n\n\n\n
With that done, here’s what my final build pipeline looks like for this project:<\/p>\n\n\n\n
trigger:\n- develop\n- master\n\nresources:\n  repositories:\n    - repository: templates\n      type: github\n      name: deltics\/azure-pipeline-templates\n      ref: refs\/heads\/develop\n      endpoint: GitHubTemplates\n\npool:\n  name: 'The Den'\n\nsteps:\n- template: gitversion.yml@templates\n\n- script: |\n    duget init\n    duget restore --path tests\n  displayName: 'Initialise DuGet'\n\n- template: build-all.yml\n  parameters:\n    delphiVersions: [ 7, 2006, 2007, 2009, 2010, xe, xe2, xe3, xe4, xe5, xe6, xe7, xe8, 10, 10.1, 10.2, 10.3 ]\n\n- template: build-all.yml\n  parameters:\n    delphiVersions: [ xe2, xe3, xe4, xe5, xe6, xe7, xe8, 10, 10.1, 10.2, 10.3 ]\n    platform: x64\n<\/code><\/pre>\n\n\n\n
I still need to specify x86<\/strong> and x64 <\/strong>builds separately but can now specify all<\/em> Delphi versions for each platform in a single step.<\/p>\n\n\n\n
You may be wondering why the <strong>resources<\/strong><\/code> section is not also needed in the build-with-version<\/strong> template.  The answer is that resources<\/code><\/strong> can only<\/em> be declared in the initial pipeline, and not in a template.  i.e. a pipeline must declare up-front any resources that any referenced templates need and that’s still the case even if the pipeline itself does not directly reference those resources itself!<\/em><\/p>\n\n\n\n
When the build actually runs, those two steps are transformed into multiple, individual build steps, one for each Delphi version<\/em>.  Each step is helpfully named in the build logs for the Delphi version and target platform, so if a change breaks this project only for specific versions of Delphi, I can see at a glance which version is affected!<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
As you can see, currently the build will abort at the first failure with any given Delphi version; subsequent build steps are not performed, unless they have a condition<\/code> applied to force them.<\/p>\n\n\n\n
I started looking at how to add such a condition and things quickly become  quite messy.  In particular, Delphi builds should only be forced if any current failure was a previously failed Delphi build<\/em>.  If there was some other, earlier failure in the build pipeline then I would not wish to force in that case.<\/p>\n\n\n\n
A tricky problem and one I may return to at some point in the future.  But not today.<\/p>\n","protected":false},"excerpt":{"rendered":"
[Estimated Reading Time: <\/span>  4<\/span> minutes]<\/span><\/span>I figured out the iterative insertion problem and my build pipeline is now TIGHT!  Fixing it was super-easy in fact, barely an inconvenience.<\/p>\n","protected":false},"author":2,"featured_media":2862,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[324,322,323,4,321,1],"tags":[326,327,292,329],"class_list":["post-2936","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-automation","category-azure-devops","category-ci-cd","category-delphi","category-devops","category-general","tag-azure-devops","tag-ci-cd","tag-delphi","tag-devops"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/www.deltics.co.nz\/blog\/wp-content\/uploads\/pipelines-hero-code-1024x256.jpg?fit=1024%2C256&ssl=1","jetpack_shortlink":"https:\/\/wp.me\/p1TKYv-Lm","jetpack_sharing_enabled":true,"jetpack-related-posts":[],"_links":{"self":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2936","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/comments?post=2936"}],"version-history":[{"count":3,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2936\/revisions"}],"predecessor-version":[{"id":2940,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/posts\/2936\/revisions\/2940"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media\/2862"}],"wp:attachment":[{"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/media?parent=2936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/categories?post=2936"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.deltics.co.nz\/blog\/wp-json\/wp\/v2\/tags?post=2936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}