Scalable Continuous Delivery Pipelines

Back when I first started building web apps we’d just “do it in production” by vi’ing Perl & PHP files on the server. This was fine because the risks and expectations were low. No big deal if I broke the app for a few hours. Good thing I made an app.php-bak copy!

As software became more critical to businesses, the risks of making changes to production systems increased. To cope with these risks we slowed down delivery through processes. Today many enterprises are so bogged down by risk aversion that they may only deploy to production once a year or less. The rate of change in businesses and software continues to increase and the expectations are even higher. Downtime is not an option but that change also needs to go out now!

We need a way to reduce risk and increase the rate of delivery. The two seem to be opposing but Continuous Delivery provides a way to deliver more often AND reduce risk.

As the name implies, the idea of Continuous Delivery is to continuously deliver changes. There are many ways to do that but the process should be scalable depending on the acceptable amount of risk. For some apps a little potential downtime is worth the tradeoff of essentially being able to “do it in production” for some changes.

Continuous Delivery can be thought of as a pipeline… There is an input / source that needs to be moved and transformed in a variety of ways until it reaches it’s output, the production system. By adding or removing the steps in between we can adjust the risk profile.

In order to have a Continuous Delivery Pipeline there are a few things necessary to facilitate any size process:

• A Source Control System (SCM) that enables developers to collaborate but also enables a direct correlation between a point-in-time in the source and a deployment of that source.
• Single directional flow from source to deployment. No more “do it in production” because that breaks the source to deployment correlation.
• A repeatable and pure method of transforming source into something that can run. Repeatability is usually accomplished by using a build tool. However, often builds specify dependencies with version ranges, sacrificing the pureness. Don’t do that.

With that infrastructure the simplest form of Continuous Delivery can be achieved. Systems like Heroku provide automated tooling for this kind of pipeline. On Heroku you can either kick off deployment by pushing a git repo directly to Heroku, which then runs the build, then stores and deploys the generated artifacts. This is the infamous git push heroku master. A newer method (which I prefer) is instead to push the changes to GitHub and have Heroku then auto-deploy those changes. Here is a demo of that:

For apps that can tolerate more risk this simple pipeline is great! But many apps need a process that better supports collaboration and risk mitigation. The next layer that may be added to the pipeline is a Continuous Integration system that runs the build and tests. If there is a failure then the changes should not be deployed. Here is an demo of doing CI-verified auto-deployment on Heroku:

Reducing risk further we could add a staging system that auto-deploys the changes (with or without CI validation depending on your needs). After manual validation the changes could then be manually promoted to production.

Taking things a step further we can hook into GitHub’s Pull Request process to deploy and validate changes before they are merged into a production branch (e.g. master). In Heroku this is called “Review Apps” in which Heroku automatically creates a fresh environment with the changes for every Pull Request. This enables actual app testing as part of the review cycle.

This full pipeline with Pull Request apps / Review Apps, CI validation, staging auto-deployment, and manual production promotion significantly reduces the risk of doing frequent deployments. Many organizations that use this kind of process are able to do hundreds of deployments every day! This also helps disperse risk over many deployments instead of accumulating risk for those big once-a-year deploys. Check out a demo of this entire flow on Heroku:

Sometimes a feature may not be ready for end user testing or launch but that shouldn’t prevent you from actually deploying the feature! For this you can use “Feature Flags” a.k.a. Feature Toggles.

Another technique that can be useful to reduce risk with Continuous Delivery is “Canary Deploys” where only a portion of users run on a new version of the app. Once a period of time validates the new version is “safe” for everyone, it can be rolled out to the rest of the users.

Of course Continuous Delivery isn’t a silver bullet as there are always tradeoffs. One of the challenges with Continuous Delivery is with database schemas. For instance, what if you were to do a schema migration as part of a Canary Deploy? With two versions of the app running simultaneously you may break one version with the schema changes from another. NoSQL / Schema-less databases are one way to address the issue. Another option is to decouple code deployments from schema migrations, utilizing testing / staging environments to validate the schema changes.

Implementing Continuous Delivery with large and complex systems can be pretty tough. But this is one of those things that if you don’t figure out how do, it won’t matter cause your business will likely fade as it is overtaken by startups that deliver software as the business needs it. If you need some more practical advice on how to get there check out my Comparing Application Deployment: 2005 vs. 2015 blog post. Let me know how it goes.