Designing Before – After Experiments

So, have you figured it out? How do we assign customers to test and control group? That’s right, randomization. So, randomization, you can match, test and control groups on all dimensions given you have a sufficient sample size. So, what does randomization mean in simple terms? So let’s take these 1,000 customers, you take all the odd numbered customers and put them as test group and take all the even numbers customers and put them as control group. That achieves randomization because the test and control groups are going to be the same for all practical purposes. But to do that, you need sufficient sample size. So what does that sample size mean? Turns out, 1,000 is this magic number.

For all practical purposes, if you have 1,000 or more customers, then you can achieve randomization. But you cannot always, in marketing experiments, reach out straight to the customers. So when would this be? For example, consider a TV advertising campaign. In a TV advertising campaign, you’re probably just selecting some cities where you turn on TV advertising, and some cities where you turn off TV advertising. In that case, you will have to select cities that are similar to each other. For all practical purposes, you don’t have access to 1,000 cities that you can just assign all odd number cities to the test and all even number cities to the control group. You’re kind of need to match these cities on known attributes.

So what does that mean? So think about you selling winter jackets. Now, you can compare sales for winter jacket between a city in Texas and a city in Maine. But you could probably compare sales of winter jackets between cities in Vermont and Maine. So for winter jackets, the known attribute was annual weather patterns. So like that, sometimes it is demographics, sometimes else it could be some other context dependent variable. So the product that you’re testing is the one that has to be determining what attributes are there on which you match the test and control group. Now we’ve seen a basic experimental design. This works in most cases.

But there is one factor that you need to consider when you’re making some more advanced experimental design. A factor that is very important in marketing is, what are the pre-existing differences between the test and control group? By some chance, are the test and control groups still different than each other even if you did randomization and matched pair assignment. So, what is the design that can take care of these pre-existing differences? That is what we call before and after design. It’s very similar to the basic design, but you just add two more steps. So how do we do this? You first still select 1,000 customers. You still say, use randomization to assign the 1,000 customers to test and control group.

Now here is the difference. In a before after design, the test and control group customers are still in the first step watching the old advertisement that was about healthy cereals. Now, you’re looking at the total sales for each of these group when they wash the oil advertisement. So it was 1,100 units and 1,000 units. Now, the test group alone then gets exposed to the new advertisement which was about taste. Whereas the old advertisement is still playing with the control group, which is about health.

You look at the sales again for the test and the control group, you see that in the test group sales was about 1,200 units and in the control group sales were about 1,000 units, because they’re still watching the same advertisement. There’s no reason to believe that suddenly sales are going to increase in the control group. But here is what before- after design reveal to you, that there was a pre-existing difference about 100 units between the test and control group, that you will have to take out from the sales lift. So the way you calculate the sales lift in a before-after design, is that you first take the difference within 1,200 and 1,000.

And from that, your takeout the difference in the preexisting condition.

That was the 100 units over here. So the difference is, in fact, 100 units because you control for the preexisting difference between test and control groups. Now, we see that with experiments, how you can manage the test and control groups. The assignment between the test and control groups and by using the before after design, which is almost like a bells and buckle design, you can control for the preexisting differences between these two groups.