Testing in Production

Testing in production is an often misunderstood concept in DevOps. People say, that’s irresponsible. You’re shipping stuff. You don’t know whether it works, and then you’re testing it. What on Earth are you doing? Well, prior to this, the process of testing was anticipating everything that might go wrong once software was released. Let’s look at how that works. Tests would run only in a testing environment. Customers would be the ones reporting production and bugs, and the test environment would need to be maintained. Because it would inevitably drift from the production environment. It also meant that you only could test for things that had happened before for the known knowns.

As Donald Rumsfeld called them, you couldn’t test for the unknowns, much less the unknown unknowns. You couldn’t anticipate things you hadn’t seen before. So as a result, you had all of these tests. You had to guess how to prioritise them. You would prioritise them, and that prioritisation would determine the coverage you’d get. And you never, never had enough time to do it right. Let’s take a look at how it’s different now. With continuous testing and telemetry to get the results from production, quality continues to be measured in the production environment. The Live Site health drives discovery of those unknowns. So in other words, if something happens that you didn’t anticipate, telemetry catches it.

That lets you not only detect and mitigate immediately. It lets you create a new test and a root cause fix to prevent the recurrence in the future. This all builds confidence in production. You use synthetic tests. That is repeated automated tests, so that you can do release over a release comparison and trending. You can use these not only to identify hard failures, but also, to identify things like performance degradation. You do the same thing in a consistent way, release over release with synthetics.

However, you also can use real user tests, where, for example, you instrument a browser or instrument a mobile device to understand how long something really takes for the user and how it performs for the user on that individual form factor. Unknown events, things you haven’t seen before, weird conditions that may come from something happening in the internet, something happening in a particular routing, something happening on a device you haven’t seen before, those are things that you can grab too. Because you have the instrumentation telemetry, and those are things that can become new tests. And then having these tests in production lets you support progressive exposure techniques, like Canary deployment, like AB experiments.

And you can use all of these observed events to inject faults into your testing process, so that you can expand the coverage of what environmental conditions you’re subjecting your software to ahead of time. So you have this feedback loop. On the one hand, you shift left to provide more testing at the right level more automatically earlier during development. And on the other hand, you shift right to improve your instrumentation and your feedback from Live Site, so that the left side gets continually stronger. That’s the idea of appropriately using testing and production.