The continuous integration pipeline evolves over time as architecture and design changes are made.   Manually configuring the pipeline can be a costly activity if the project contains a large number of modules.   Setting up a ‘Continuous Pipeline Configuration’ process may become a necessity for reducing this time-consuming effort, as well as for reducing the risk of important bug fixes or other changes being left out of the final product build due to an improper pipeline configuration.

Developing a CPC process to regularly validate and adjust the pipeline is something I highly recommend for any size project, but especially for large, complex modular projects. CPC is actually quite easy once you know what kind of source code changes will require corresponding changes in the pipeline configuration.

The CPC process works much the same as a CI server watching a version control system for change.   The pipeline configurator watches for changes to the build dependency graph for the full product build and makes the appropriate pipeline adjustments to the build and test triggers in the CI server. When a build or test dependency is removed from the graph, the corresponding CI trigger is removed and when a dependency is added, a trigger is added.

Continuous Pipeline Configuration

1. Capture the product build and test dependency graph
2. Capture the existing Continuous Integration pipeline configuration
3. Compare the dependency graph to the pipeline configuration
4. Update the CI triggers as needed.

The product build and test dependency graph is the complete list of ‘A depends on B’ relationships.   I like to think of this as the producer-consumer relationship where B is the producer and A is the consumer if A depends on B. Both A and B can be any type of process, activity or event, such as build, test, bundle, or deploy. It is not mandatory for these producers to actually publish binary artifacts. In some cases, the only output is a pass or fail result – enough to determine if a corresponding CI trigger should or should not execute the next event in the pipeline.  If you are using the Artifactory for your artifact repository, you use it to store and retrieve the dependency graph via the Artifactory Query Language.

If the consumer module’s dependency is a static constraint, the dependency is a ‘pull’ connection in the pipeline and if the dependency is a dynamic constraint, the dependency is a ‘push’ connection. Pull connection don’t need a CI trigger since the change to pick up new versions will be a code change that triggers the ‘consumer’ build or test. Push connections require finished build triggers in the CI server so new ‘producer’ builds will trigger ‘consumer builds if they succeed.

Continuous Pipeline Configuration is not limited to Continuous Integration.  It can be used for configuring of all of the pipelines in the Continuous Delivery process.  My thoughts on CD are very similar to those of Jame Betteley as he describes his views here: Methods and Tools.  Continuous Delivery extends the core principles of CI all the way through to the end of the software development life cycle.

We’re automation experts.  Let’s take Continuous Integration – and ultimately Continuous Delivery – to the next level with Continuous Pipeline Configuration.

Software configuration management systems must be designed for change.  This need is easily recognized by the simple fact that software organizations are constantly in a state of transition.  There are requirement changes dues to organizational growth and SCM Stakeholder Transitions, plus rapidly changing technology advancements are driving the need for SCM changes.

Some really cool plugins have been developed to help with this change, but I’m averse to using them for reducing short-term configuration time by tightly coupling two parts of the SCM system together.  This is often done at the expense of making future changes more difficult.

One example is utilizing a continuous integration server plugin for publishing builds, such as the TeamCity-Artifactory plugin.  This plugin will save you from having to include artifact publishing as part of the build scripts, but you will have some redo work if you want to change the CI server.  This kind of rework means it could be much more expensive to change integration systems when a new product comes along, or if your company decides to cut expenses by switching to an open source system like Jenkins or Travis CI.  The SCM design is likely to more flexible, at least from a CI server perspective, if a build technology plugin, such as the Gradle-Artifactory plugin is used for publishing.

Another way to keep your SCM system changeable is to avoid engineering build scripts to be dependent on code stored in a specific version control system.  One way this can happen is if the build script retrieves the revision number and uses it as the build number.  This will complicate a migration from one VCS to another at some point in the future.  A better design is passing the revision number to the build script as an argument – making the source code tree version control agnostic and easier to move.

Just as with code modules, reduced coupling between the underlying technologies in the SCM design can simplify re-engineering when changes are needed.

SCM is Changeable