Generalisability and limitations of findings

By the end of this step, you should be able to define bias and confounding, describe the differences between the two main types of bias– selection bias and information bias, appraise how bias affects different types of study, and assess the impact of confounding and how it can be minimised. Bias means deviation from the truth. And bias in a study affects the results. Bias may be due to the study design or the way in which studies carried out. We cannot always measure the extent to which bias influences the study’s findings. Bias cannot be easily corrected for by increasing the sample size, but sometimes can be accounted for in the analysis.

When we consider a study results, we should always ask if the observed effect could be due to bias. There are many, many types of bias. But don’t worry. You don’t need to know the names of them all. They can really be put into two main groups– selection and information bias. In addition, confounding is another type of bias, which we will discuss later in this step. Selection bias occurs when there is an error in the process of identifying the study population. Selection bias can affect the observed results. For example, in a cross-sectional study like a population based survey, asking for volunteers rather than using a random selection from a population can result in selection bias.

People with an interest in a topic are more likely to volunteer. And these people may not represent the general population. As we’ve heard in previous steps, usually, we select a sample of the population of interest rather than complete a full census. However, incomplete coverage of the population will lead to over- or underrepresentation of the eligible participants in the sample. For example, let’s consider how to estimate the prevalence of disability in Vanuatu. If we only look for people with disabilities in clinics or hospitals, this will lead to selection bias. We will fail to count many people with disabilities, as they may not have access to health services. A better sampling frame is the population.

Remember that the response rate is the proportion of eligible participants in the sample who are available for the survey. A good response rate in cross-sectional studies is essential. Otherwise, the results will not be representative of the population we are studying. It is vital to manage and minimise non-response in a cross-sectional study. We can do this, for example, through repeat visits or by involving community leaders to help sensitise people in advance. Case control studies are particularly prone to selection bias. We can minimise bias through careful selection of cases and controls in this kind of study.

For example, selecting people with disabilities from a hospital or clinic for inclusion in a case control study would most certainly result in selection bias compared to selecting randomly from people with disabilities in the community, as those attending clinic may be different from those in the community. In a well-conducted randomised controlled trial, selection bias is eliminated by randomization, as long as loss to follow up is minimised and the randomization procedure is carried out properly. Note that if we allocate the treatment or intervention without randomization, this will result in selection bias. An example of this would be the allocation to the intervention or control by the order in which patients come to the clinic.

For example, this might result in bias because people with mobility difficulties might take longer to reach the clinic and, therefore, all end up in the same arm of the trial. Information or measurement bias occurs when there’s any systematic error or flaw in how we collect data. This can be due to error by the data collector or within the instrument used. And the resulting misclassification can be either random or non-random. Let’s show this through two quick examples. Incorrect calibration of a piece of clinical equipment, such as an audiometer to measure hearing loss, can lead to a non-random misclassification. For example, by subtracting the same amount from each measurement in each participant.

Systematically, then, each participant’s result is misclassified in the same way. Random misclassification could occur if a data collected assigns the response of the Washington group disability questions, rather than asking the participant to self-report. This brings in subjectivity to the assessment, and may vary from data collector to data collector. These types of information bias can be reduced by repeat or automated measurements and calibration of any instruments used. The use of validated tools that are known to be relevant to the population of interest, good quality training of field workers, and use of protocols to ensure tools are always used systematically. Recall bias is a type of information bias and is a particular problem in case control studies.

It occurs when participants report or recall information. For example, information on livelihoods in a disability study in Vanuatu was obtained through asking participants to describe activities they spend most of their time doing across paid roles and household tasks.

Underreporting the amount of time spent in different activities may reduce the association of disability with time spent on productive activities. Overreporting gives a greater estimate of the association between disability and these outcomes. In summary, bias can lead to departure from the truth. That is the true value of the prevalence or incidence of the condition of interest in the population or the association between a disease and exposure or the effect of an intervention. We minimise selection bias in our participant selection through well-devised sampling and management of non-response. We minimise information bias through equipment calibration, use of validated tools, good quality training, and protocols. The degree to which bias influences the study’s findings cannot always be measured.

However, we can reduce the effect of bias by paying close attention to how a study is designed and carried out.

Confounding is a form of systematic bias. The word confounding comes from Latin and means to mix up. A useful definition of confounding is a nuisance to be prevented or ruled out.

A major part of research, particularly epidemiology, is establishing associations between the exposure and the risk of disease. Confounding occurs when the association between exposure and a disease is mixed up with the effect of another exposure. This leads to the relationship between the exposure being investigated and the disease being distorted by another hidden factor. Confounding occurs when the effects of two associated exposures have not been separated. The impact of confounding can lead researchers to make an incorrect conclusion that is in effect due to one exposure rather than another. Let’s look at an example. We know that age is a risk factor for COVID-19 mortality. We also know that disability is a risk factor for COVID-19 mortality.

To look at the risk of COVID-19 deaths associated with disability, we need to take away the effect of age, as this confounds the association we are looking at between COVID-19 and disability. Confounding is concerned with alternative explanations for the associations seen between the exposure and outcome. Exposure can be an known or possible risk factor, for example, having a disability. Outcome can be a disease or condition. In this example, mortality from COVID-19. We need to remove the influence of a confounding factor– in our example, age– in order to get nearer to the truth about the direct relationship between disability and COVID-19. We can do this in a couple of ways.

We need to remove the influence of a confounding factor in order to get to the truth. We can do this in a couple of ways. Through our choice of study design. Ideally any confounding is taken care of in the design stage. Or by adjusting, also known as controlling, for the confounding factor in the statistical analysis. For a risk factor to be regarded as a confounding, the rules are it must be associated with the exposure being investigated, and it must be independently associated with the disease or outcome being investigated. It must not be on the causal pathway. This is a factor that comes on the route from exposure to disease.

Let’s go back to our example– looking at the risk of COVID-19 deaths associated with the exposure disability. Remember that we are interested in the relationship between disability, the exposure, and COVID-19 mortality, the outcome. Age is associated with both the exposure and the outcome. Age is not on the causal pathway between disability and COVID-19 mortality, meaning that having a disability doesn’t lead to ageing, which leads to COVID-19 mortality. Therefore, age is a confounder because it is independently associated with disability and COVID-19 deaths, and is not on the causal pathway. Given this, if we want to look at the risk of death associated with disability, we have to take away the effect of age.

Another important point to remember is that confounding depends on the relationship of interest. If instead of disability we were interested in the relationship between age and COVID-19, disability would be the confounding here. One way we can take away the effect of age is in the design stage by stratification. In stratification, we adjust our sample to ensure that we have enough participants in each different category of the confounding variable. This means that in analysis, we can compare the risk of COVID-19 deaths in people with and without disability in study participants of the same age.

The other way is to adjust for age in the statistical analysis– for example, by carrying out a stratified analysis or more complex analysis such as multi-variable regression modelling. Before conducting a study, it is important to think about what the potential confounder confounds could be in your study of interest. This might involve searching the literature for studies on your particular topic area. When designing your study, you must make sure to measure the potential confounders identified so that you can account for them. In summary, confounding is a systematic bias that can seriously distort the findings of observational studies. Confounders are independently associated with the outcome being studied, but not on the causal pathway.

The effect of a confounder can be minimised in the study design or in the statistical analysis. We must try to control all known potential confounders in studies, although we may not be able to address them all completely.