Skip main navigation

Continuous Testing

Let’s talk about continuous. Testing continuous testing is this practise of not just testing between dev and prod as its own independent thing, but testing throughout the entire lifecycle. So what is continuous testing? Well, first of all, it’s the execution of the test repeatedly against a code base in some deployed environment. I’m going to include the build environment as well, but then also against this deployed environment. We’re talking about automated unit integration tests, coded user interface tests, load tests. All of these things should be common. Now, as we get closer, and closer, and closer to production, very often, the depth of those tests increases.
So when we’re in a build environment, we’re going to do– we’re going to run all of our unit tests. And we’re going to deploy it out to a environment. And maybe we’ll run some smoke tests, but not our full regression suite. Maybe we’ll wait till we get one step further to run the full regression suite. Again, this depends on how quickly you’re pushing things into that dev environment and how quickly you’re pushing them into your test environment. Ideally, if you’re moving slowly anyway, you might as well run all your suite of automated tests at every push into dev. But sometimes, that’s not possible, because you’re just moving too fast.
So in practise, continuous testing and keeping those test suites up to date is one of the most difficult and challenging parts of a good, continuous delivery pipeline. It’s difficult to get right. Where does this continuous testing fit inside of kind of a flow? Looking at the deck, you’ll see going from developer when they check into version control, that will trigger a build in a unit test, and we’re going to run all of our unit tests here. Make sure that it builds, it compiles. Then we move to automated acceptance testing. If all of those passed, some automated user acceptance testing. Maybe a manual step in there. Not for sure, and then off into release it well as well.
And I’ve coloured that on the slide a little bit differently. A slightly different colour on the release, because this is where we’re not just going to be running some synthetic tests, which are the tests that we have built to define to run maybe even in our test environment. But we’re going to be gathering metrics, telemetry from real user metrics. We’re going to be gathering that telemetry from release, and we can find out if something’s going wrong that potentially we didn’t catch earlier. Now, I’m going to throw in one additional tidbit into this, and that’s the goal. Some of this goal of telemetry and automated testing are– what we’re focused on is we want to minimise the mean time to repair.
That means we’ve got to detect the problem quickly and fix it very, very rapidly. This is more important to the DevOps environment than the mean time between failures. Now, we’re all geared towards thinking mean time between failure. Because when we took our product and we shipped it, we handed it to a customer. We absolutely positively couldn’t have bugs in that thing, because they’re very hard to repair. Now, however, with this continuous delivery mindset, where we’re deploying potentially to servers or we’re able to pull those changes down into client applications rapidly, we’re able to instead fix bugs very, very rapidly. Preferably before the user discovers that they’re even there.

Steven Borg returns to introduce us to Continuous Testing, which is the repeated testing against a code base and a deployment environment.

Continuous Testing workflow

This article is from the free online

Microsoft Future Ready: Fundamentals of DevOps and Azure Pipeline

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education