Published July 30, 2023
by Doug Klugh
Bonus Testing
What if I told you that you can test your software in production? Well, you can with the right testing strategy and correct deployment methods. A DevOps approach to delivery will enable your team to conduct limited manual testing of new functionality, extensions, and patches in a production environment before releasing to your customers. Although it should never replace the unit, service, and system-level testing in your lower environments, it can help to ensure that your software will run as expected in production.
Following are deployment methods that provide three different ways of mitigating risks associated with releasing new software versions by testing new features, enhancements, and patches in a production environment.
Blue/Green Deployments enable teams to deploy software to an inactive production (green) environment in order to test on that production platform. Once testing is complete and the software is ready to be released, your team will simply redirect traffic from your live production (blue) environment over to the green environment. This can be done during normal business hours with zero downtime and will help to mitigate risks associated with production releases while greatly reducing deployment time.
Ring Deployments are used to gradually release to select groups of users. Starting with small groups of low-risk users, then gradually expanding the user base to include higher risk users, your team can limit the blast radius in the event of something going wrong and contain the impact to your most loyal customers. Be sure to assign users that are most tolerant to failure to the inner most rings, then assign higher-risk users as you move outward. For example, you could assign innovators to the inner ring, early adopters to the second ring, then all other users to the third, most outer ring. Of course, if you need to split your users further, you can always add more rings. This method not only provides an opportunity to test new features in a production environment, but also enables your team to gather telemetry, gauge user response, and attain customer feedback.
Canary Releases manage the risk of releasing new software by first releasing to a small user population, then gradually rolling out to the entire user community. If problems are detected, users can be quickly re-routed back to the old version. In addition to functional testing, this can be used to facilitate capacity testing in a production environment with a dependable rollback strategy. If quality is not an issue, canaries can be used for A/B testing of hypotheses of user engagement. Canary Releases should not be used on critical systems that cannot tolerate failure.