Published November 13, 2022
by Doug Klugh

Fail Fast

Build a suite of tests that is optimized to run very quickly that serves as the entry gate to the deployment pipeline.  In less than ten minutes, the commit stage should either create deployable artifacts or eliminate builds that are unfit for production and notify the team that the application is broken.  This suite of tests should include all unit tests, along with a small selection of other types to ensure that the most common failures are caught early.  None of these should execute via the user interface to safeguard against Fragility and lengthy execution, while promoting Defect Localization.

Our goal is to spend the least amount of time and effort to determine if a build is broken and provide rapid feedback on the most common failures a developer is likely to introduce into the codebase.  Once a developer commits code to the source code repository (repo), a series of automated steps is initiated that includes compiling and linking the code, running static code analysis, executing the commit test suite, and creating the binaries and other artifacts.

A failure in any of these steps will fail the commit stage, preventing any further testing, integration, or deployment.  Even with a failure, the commit stage should run to completion in order to provide an aggregated report of all the errors so they can all be fixed at the same time.  The commit stage should only be halted if an error prevents the remaining steps from running, such as a compile error.

The catalyst to automated integration and deployment is when a developer checks code into the repo.  But before that happens, the developer must first create the build and run the commit tests locally.  Once that succeeds, he/she must pull the latest changes from the trunk and merge them with his/her local changes, then rebuild and rerun the commit tests locally to ensure his/her changes work with everyone else's changes without introducing any new errors.  This will help to distinguish a developer's bug from a bug introduced with the merge.  Once that succeeds, the developer will push his/her changes onto the trunk (to share the changes with the rest of the team), which will initiate the automated integration and deployment process, beginning with the commit stage.

Mature software testing usually includes unit testing to verify implementation, component testing to validate functionality, and integration testing to verify, well...  integrations.  But none of those tests do anything to verify quality attributes — which are sometimes referred to as a subset of non-functional requirements.  While compiling, building, scanning, and testing go a long way to validate that your application works, they do very little to ensure you have a healthy, high-quality codebase.

Along with functionality, your commit tests should verify preset thresholds for qualities such as test coverage, amount of duplicate code, number of warnings, size of functions, classes, and components, Cyclomatic Complexity, and afferent and efferent coupling (see Coupling Metrics in my post Principles of Component Coupling).  Failure to meet these thresholds should fail the commit stage the same as any functional test.

A commit stage that successfully passes all functional gates can often be interpreted as a false positive, suggesting that the quality of the code is good when, in fact, it is not.  A mature development team will want the commit stage to fail if, for example, the unit test coverage falls below some threshold (say 90%).  Or if there are too many warnings, too much duplicate code, too much complexity, functions that are too big, or too many unstable classes — indicating too much coupling.

Keep in mind that when the commit stage fails, the team should pull the Andon Cord, stop whatever they're doing and immediately fix the problem.  If the problem cannot be fixed within a predetermined amount of time (say 10 minutes), roll back the changes.  Make sure that everyone agrees on the functional and non-functional quality gates that are included in the commit stage.  Otherwise, team members will stop taking failures seriously and your CI/CD process will quickly fail.

Related Posts