Continuous Delivery as done in Colours

2 years ago I posted an article on HabraHabr, which describes how Continuous Integration is done in Colours, where I am working ATM. Those time it was published only on Russian, but, it seems to me, that newest explanation should be published on both languages (russian version is accessible here - http://habrahabr.ru/post/249523/).

Our Continuous Delivery process (further referred as CD) have 3 pillars underneath:

1. Source control system
2. Branching model, in which each base branch are representing application state on each target server (QA, UAT and production respectively)
3. teamcity as build server.

Our CD solution proposes following to our developers and testers (I would cover all this later on):

• Deployment through ssh to Linux-driven servers
• Deployment by means of MsDeploy to Windows-driven servers (currently, we have only one limitation - server OS should be at least Windows 2008 R2)
• Automatic maintaining of internally used and open-source assemblies developed by us in forms of nuget packages on respective nuget servers (for internal use - on internal nuget gallery, for open-source - on Nuget.org) along with publishing sources on respective SymbolSource servers (there is internal one for internal nugets).

SSH deployment

This one is simplest one, as ATM we are using only PHP-driven projects - thus, delivery is simple: get sources from source control and upload them via scp to server. There is a Teamcity plugin for this - 'deploy-runner', which is covering all holes. At current moment, those build are used only for internal testing. In future, they will receive integration and unit test, and, some day, we will be using them to complete whole application lifecycle.

Nuget, SymbolSource, packaging and deployment

Inside of our corporate network we have deployed local Nuget gallery (forked from https://github.com/NuGet/NuGetGallery) and SymbolSource server for this gallery (http://www.xavierdecoster.com/setting-up-your-own-symbolsource-server-step-by-step). As this is also webapplications - Teamcity covers theirs updates by watching our forks and delivering changes, using described further steps.

Also Teamcity is responsible for Nuget packages delivery to respective gallery (internal packages are delivered to internal gallery, open-sources one - to Nuget.org) and publishing sources of those packages to respective SymbolSource server, by utilizing integrated into Teamcity tooling.

msbuild, MsDeploy and everything around…

Just several words about MsDeploy and reasons for moving out of SSH driven delivery (initially, idea was to create unified delivery, which would cover both Linux and Windows web applications). Despite of the fact, that SSH driven deliveries are pretty fast, but preparation requires a lot of time and expensive HDD operations. Additionally to this fact, in SSH driven delivery for .net web application we have no way to influence size of published package and only way to deliver – was to delivery whole web application. But things will be totally different as soon as you take MsDeploy, which itself makes a decision: should this particular file be included in delivery package, if it is changed, or it should be deleted on target application and so on.
So, to guarantee fast and reliable delivery – we've decided to stick with MsBuild engine and MsDeploy. To control deployment process we developed several projects (targets files) to be imported in project to be built. They are unified by type of procedure and differs only by project type (Sitecore, Umbraco, Episerver or just not CMS driven web application). Next to this, I would cover those targets a little bit more…

Sitecore

Sitecore.targets file covers these steps:

1. Prepare application to deliver logical part of it (database changes to be delivered by Sitecore itself or by pluggable 3rd party applications) – delivery of configuration file, update of CI assembly. As this step requires application restart (to read changes) – it is to be called only in case we have something to deliver to database
2. If first step is called – this step also invoked – get up application by requesting first page of it
3. Static web.config transformation (e.g. one for all web application) to ensure that Sitecore would cleanup itself after Sitecore packages deployment (our code implementation is placing them in temp folder)
4. Import of shared targets, which is covering in-build steps to be done. Order of execution is covered by AfterTargets and BeforeTargets directives in this file.
5. If we chosen to keep files on server, even if they are removed in IDE (e.g. MsDeploy parameter SkipExtraFilesOnServer equals to true, while it defaults to false) – these targets file also dictates which files should be definitely cleaned up on target web application (use case: huge code refactoring with assemblies rename – if bin is not removed – .NET would generate ‘Ambiguous reference' error).

Episerver

Episerver.targets covers these steps:

1. Adds latest version of CI assembly
2. Import of shared targets, which is covering in-build steps to be done. Order of execution is covered by AfterTargets and BeforeTargets directives in this file.
3. If we chosen to keep files on server, even if they are removed in IDE (e.g. MsDeploy parameter SkipExtraFilesOnServer equals to true, while it defaults to false) – these targets file also dictates which files should be definitely cleaned up on target web application (use case: huge code refactoring with assemblies rename – if bin is not removed – .NET would generate ‘Ambiguous reference' error).

Umbraco

Umbraco.targets covers these steps:

1. Adds latest version of CI assembly
2. Import of shared targets, which is covering in-build steps to be done. Order of execution is covered by AfterTargets and BeforeTargets directives in this file.
3. If we chosen to keep files on server, even if they are removed in IDE (e.g. MsDeploy parameter SkipExtraFilesOnServer equals to true, while it defaults to false) – these targets file also dictates which files should be definitely cleaned up on target web application (use case: huge code refactoring with assemblies rename – if bin is not removed – .NET would generate ‘Ambiguous reference' error).
4. Update of application version in ClientDependency.config to make sure, that user would on operate on stale cached in browser css and js.

Build itself

All builds are unified and templated in TeamCity (to ease management and additional steps integrations) and could differ only due to CMS (those differences are covered by targets files imported) or by customer/hosting requirements. Further I will cover all custom developed steps in order of execution (in brackets at start of description I would add targets file name in case, if these step is specific only for this particular target).

1. Informational email to be sent to manageable email addresses, which contains following information:
   - to which project and environment belongs this particular starting build
   - who started the build
   - changes, included in build (comments to check-ins in repository. If comments begins from #number – #number becomes a link to request in our project management system).
2. (Sitecore) Check, if there is some packages (logical parts of content databases of Sitecore) to be delivered. If they are present – steps 1 and 2 from Sitecore.targets is invoked
3. (Sitecore) Packages delivery with WCF service
4. (Sitecore) Publish with WCF service
5. (microsoft Network Load Balancer) If application is using several servers and traffic is balanced with help of MS NLB – this step would remove server from NLB with Drainstop (no new users would arrive to server, application would serve only remaining users. As soon as remaining users left – node is stopped)
6. App_offline.htm delivery to application webroot (to deliver end users pretty informational message, that application is being updated) and switching monitoring service off (as application would return HTTP status code 503 with App_offline.htm in webroot). Which monitoring services is involved – controlled by build parameters.
7. Nuget restore – though, normally, nuget restore is switched on on solution level, still, TeamCity would generate error, in case team is using Nuget's which changes build process (for example, Microsoft.Bcl.Build)
8. MsBuild publish – everything ‘magically' happens here…
   I will not cover what is done by MsBuild itself, just describe changes to build process added by sharedtargets.targets, which is imported by web application specific targets file.
   1) If we need to add CI related assembly – it is added here, by marking it as Content with meta data CopyToOutputDirectory=Always
   2) AddVersionTxt – adds version.txt to webroot, which contains build number and date (it eases understanding, what is delivered currently to our testers)
   3) (Sitecore) Step 3 from sitecore.targets file is called here (static web.config transform)
   4) Encrypt – if in application repository to configurable path RSA key is added and msbuild parameter EncryptWebConfigOnTeamcityBuild is set to True, then TeamCity would encrypt appSettings and connectionStrings sections of web.config (additional sections could be added on project level)
   5) If application is using additional database (or additional custom tables in CMS databases) – then we are calling SqlUpdaterSupportTarget. It acts pretty the same as second build step – checks, if in configurable path there was some changes introduced and if yes, updates configured in TeamCity database. Currently, it is working with MSSQL and MySQL (MariaDB), but have one limitation – works only with one additional database.
9. (EpiServer) As Episerver is storing CMS related files out of webroot (by default, this fodler is called AppData) and accesses them via Virual Path Provider – we need to check, if there was some changes in this folder – and if they are present – deliver them by means of MsDeploy
10. Delete App_Offline.htm, get web application up and enable back monitoring services.
11. Wget visits first level pages of web application on sites, which are passed in build configuration.
12. OWASP analyzes published web.config, and generates report, which is accessible in TeamCity build reports tab.
13. Some standard web application checks and email to manageable email addresses (same as in step 1) with checks and build results to group.
    ATM, standard checks are just checking presence and correctness of robots.txt (with this parser https://code.google.com/p/robotstxt/) and sitemap.xml file.
    This step will expand in future to include all kinds of integration and unit tests.

Besides web applications, we also develop additional Windows services to support our web applications (or 3rd party ones). And, naturally, we should deliver them also…
This is covered by separate build template, which uses following steps:

1. Nuget restore
2. Visual Studio build in Teamcity
3. Stop and remove running services on target server with help of runCommand MsDeploy
4. Files delivery with MsDeploy
5. Reinstallation of service on target service with MsDeploy runCommand provider

One more build is required by TeamCity to self-service: build and update tools/files used in described previously. It consists of next steps:

1. Nuget restore
2. Visual Studio build
3. powershell script to deliver build results to each Teamcity build agent

Besides those builds, covering code delivery (in one way or another) there is also one TeamCity sevice build template, which is accompanying delivery builds. This particular “build” consists out of 2 steps and have only one target – recycle web application. It is required to allow application restart without access to server and consists of following steps:

1. MsBuild recycles web application with help of recycleApp provider, which is integrated in MsDeploy
2. Wget visits first page of web application in spider mode, to get application up.

This build is not called much, as it is last resort.

Working on…

As final touch, I used open-source tool, called Gource, to visualize CI repository timeline and demonstrate it: