Skip main navigation

Testing in Production

Testing in production is an often misunderstood concept in DevOps. People say, that’s irresponsible. You’re shipping stuff. You don’t know whether it works, and then you’re testing it. What on Earth are you doing? Well, prior to this, the process of testing was anticipating everything that might go wrong once software was released. Let’s look at how that works. Tests would run only in a testing environment. Customers would be the ones reporting production and bugs, and the test environment would need to be maintained. Because it would inevitably drift from the production environment. It also meant that you only could test for things that had happened before for the known knowns.
As Donald Rumsfeld called them, you couldn’t test for the unknowns, much less the unknown unknowns. You couldn’t anticipate things you hadn’t seen before. So as a result, you had all of these tests. You had to guess how to prioritise them. You would prioritise them, and that prioritisation would determine the coverage you’d get. And you never, never had enough time to do it right. Let’s take a look at how it’s different now. With continuous testing and telemetry to get the results from production, quality continues to be measured in the production environment. The Live Site health drives discovery of those unknowns. So in other words, if something happens that you didn’t anticipate, telemetry catches it.
That lets you not only detect and mitigate immediately. It lets you create a new test and a root cause fix to prevent the recurrence in the future. This all builds confidence in production. You use synthetic tests. That is repeated automated tests, so that you can do release over a release comparison and trending. You can use these not only to identify hard failures, but also, to identify things like performance degradation. You do the same thing in a consistent way, release over release with synthetics.
However, you also can use real user tests, where, for example, you instrument a browser or instrument a mobile device to understand how long something really takes for the user and how it performs for the user on that individual form factor. Unknown events, things you haven’t seen before, weird conditions that may come from something happening in the internet, something happening in a particular routing, something happening on a device you haven’t seen before, those are things that you can grab too. Because you have the instrumentation telemetry, and those are things that can become new tests. And then having these tests in production lets you support progressive exposure techniques, like Canary deployment, like AB experiments.
And you can use all of these observed events to inject faults into your testing process, so that you can expand the coverage of what environmental conditions you’re subjecting your software to ahead of time. So you have this feedback loop. On the one hand, you shift left to provide more testing at the right level more automatically earlier during development. And on the other hand, you shift right to improve your instrumentation and your feedback from Live Site, so that the left side gets continually stronger. That’s the idea of appropriately using testing and production.

To connect testing and DevOps, it is important to set the stage by starting with the idea of shifting left (and shifting right). These concepts encompass more than just testing, they also encompass the entire development cycle.

This article is from the free online

Microsoft Future Ready: Fundamentals of DevOps and Azure Pipeline

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education