1 Introduction
Epidemiological studies can provide valuable insights about the frequency of a disease, its potential causes and the effectiveness of available treatments. Selecting an appropriate study design can take you a long way when trying to answer such a question. However, this is by no means enough. A study can yield biased results for many different reasons. This article explores some of these factors and provides guidance on how to deal with bias in epidemiological research. We will learn about the main types of bias and what effect they might have on your study findings. We will then look at the concept of confounding and will explore various methods to identify and control for confounding in different study designs. In the last section we will discuss the phenomenon of effect modification, which is key to understanding and interpreting study results. We will finish with a broader discussion of causality in epidemiology and we will highlight how you can decide whether findings indicate a true association and if this can be considered causal.
2 Validity and Bias
When critiquing epidemiological studies, you will often hear or =read about concepts such as validity and bias which determine whether the results of a study are relevant and should be trusted or not.
When critiquing a particular study, there are some key questions that you would consider. One of these is whether any inferences arising from it are valid for the source population of this study. For example, a study may report an association between a new drug and improved survival among male cancer patients in a university hospital. There are many reasons why this could not reflect the truth such as flaws in the design or the execution of the study. But if we believe that this association truly exists among this group of patients, then we say that this is a study with internal validity.
Another equally important question is whether these inferences are applicable to individuals outside the source population. Internal validity is a prerequisite for this. If we don’t think the results reflect the truth in the source population, discussing if they can be generalized to other groups of people is pointless. But let’s assume that taking this new drug is in fact associated with improved survival among male cancer patients in the university hospital where the study was conducted, and the researchers have done an excellent job showing this. We would say that the study has external validity if we believe that this finding can be applicable to other groups of cancer patients, female patients in the same hospital or patients treated in different settings and countries.
External validity sometimes referred to as generalisability and largely determines the real life impact of a certain finding beyond the specific setting where the research was conducted. Closely linked to validity is the concept of bias. Simply put, an inference is valid when there is no bias. According to one popular definition, bias is any trend in the collection, analysis, interpretation, publication, or review of data that can lead to conclusions that are systematically different from the truth. The key word here is systematically. A systematic error in the design and conduct of the study can result in bias which means that the observed results may be different from the truth.
In conclusion, systematic error can introduce bias in the study which in turn hurts its validity. Bias can take many forms, and scientists have identified many types of bias and their variations over the years. To make things more difficult, there are myriad different classifications and names for bias observed in epidemiological studies. We will consider three broad categories of bias:
- Selection bias
- Information bias
- Confounding
2.1 Selection bias
One of the main categories of bias in epidemiological studies is selection bias. In practice when doing research, it is almost impossible to examine the entire study population. This is why we select the sample. Despite our efforts to select a sample that is representative of the study population, it may happen that an individual’s chance of being included in the study sample is related to both the exposure and the outcome. When this happens, we get a biased estimate of the association between the exposure and the outcome and we say that there is selection bias.
Let’s consider a case control study where the exposure is diet rich in red meat and the outcome is colon cancer. As we’ve discussed, our sample will include only a fraction of the study population. In one scenario, people with colon cancer have a 60 percent probability to be included in the study sample while people without colon cancer have a 40 percent probability to be included. Clearly, the disease status is associated with inclusion in the sample but within its disease category, individuals are equally likely to be selected regardless of whether they eat a lot of red meat or not. In this case, there is no selection bias.
In another possible scenario, the disease could be irrelevant with regard to being included in the sample. However, individuals eating a lot of red meat could be less likely to be included in the study compared to those not eating red meat. For example, because part of the recruitment strategy was to place posters in shops for vegetarians. In this case, the probability of being included in the sample is associated with the exposure eating red meat but not with the outcome which is colon cancer. Therefore, there is no selection bias in the study.
So, when do we have selection bias? Consider the same case control study. This time, 60 percent of people with colon cancer accept to participate regardless of their diet. Among people without colon cancer, 50 percent of those who eat red meat and 40 percent of those who don’t eat red meat decide to participate. In this scenario, participation in the study sample is associated with both the exposure and the outcome. Therefore, it is a typical case of selection bias and our estimate will be biased.
It is not a coincidence that we have used case-control studies in this example, Case-control studies are particularly susceptible to selection bias but there are ways to minimize selection bias, we will mention three of them. First, researchers try to select controls which are representative of the study population in terms of exposure to the factors under study. Also, in all study designs, it is important to keep non-response to a minimum. When many people decline to participate, it becomes more likely that some bias could be introduced. Finally, it is always good practice to compare those included in the sample with those who declined to respond and explore whether there are any systematic differences. Selection bias can seriously undermine the validity of the study, it is therefore really important that you take this into account when designing or critiquing epidemiological research. Of course, there can be other sources of bias as well.
2.2 Information bias
Much like selection bias, information bias has many different names and subcategories, but includes misclassification of the exposure or the disease status or both. Let’s consider an example of a case-control study which aims to look at a potential association between smoking and lung cancer. Regarding exposure, we would obviously need to assess whether participants were smokers or not and how much they smoked. We would also need to classify people as having lung cancer or not, as this is the outcome of interest. Both exposure and outcome could be misclassified. For instance, some heavy smokers may be erroneously classified as light smokers or some lung cancer patients may not receive the correct diagnosis. Usually this happens either because the study variables are not properly defined or due to flaws in data collection.
Let’s examine some of these flaws more closely. One common flaw in data collection occurs when interviewers ask individuals about their exposure status. In our example, interviewers would ask individuals with and without lung cancer, if they have been smoking. But the interviewers might be more thorough in assessing past smoking when interviewing people who have been diagnosed with lung cancer, exactly because they expect that lung cancer patients are likely to have been smokers. This would lead to misclassification of exposure status and eventually to a biased odds ratio. This type of information bias is called Interviewer bias. Luckily, this can be prevented if the interviewer does not know the disease status of the individual or if the collection process has been carefully standardised, so that interviewers follow a strictly defined protocol when they collect data from participants. However, interviewers are not the only potential source of information bias.
When patients with lung cancer are asked to report whether they have smoked in the past, they might be more likely to recall a brief period of smoking along time ago compared to those who don’t have lung cancer. This is not unexpected. Our memory is not perfect and we often forget things that have happened in the past. But when we get sick, we try hard to remember any details that could be linked to our disease. Details that we would otherwise erase from our memory. This phenomenon is called Recall bias and is a common type of information bias. We can prevent it by using objective ways to assess exposure such as medical records or biomarkers. We should highlight that Recall bias specifically refers to the differentially inaccurate recall of past exposure between cases and controls.
When all the participants have trouble remembering their exposure status, but this has nothing to do with their disease, there’s no recall bias. This is a principle that can be generalised, when exposure status is misclassified but equally so among cases and controls, we speak of non-differential misclassification. The same term applies when there are errors in determining the outcome, but they occur equally among exposed and non-exposed individuals. When non-differential misclassification occurs, the odds ratio we obtain is biased always towards the null. In contrast, misclassification is differential when errors in determining an individual’s exposure status occur unevenly among cases and controls or when there are errors in the diagnosis of the disease which occur unevenly among the exposed and non-exposed individuals.
Differential misclassification also leads to a biased estimate, but we cannot predict if it is biased towards or away from the null. As we can see, on all these occasions, there is information bias that could lead to a biased estimate. We have seen how these can influence the results of your study and with ways to prevent this. Together with confounding, which we will explain later, the broad categories of selection and information bias can explain essentially all the issues that could undermine the validity of a study.
3 Association and Confounding
3.1 What is Confounding ?
Correlation does not imply causation, one of the reasons we say this is confounding. Consider you are studying Down syndrome, and you come across a graph, which clearly shows that there is an association between birth order and Down syndrome. It seems there’s a higher risk of being born with Down syndrome among children with higher birth order. Now the question is, is it the birth order that increases this risk? You suspect that there maybe another variable correlated with birth order, which is responsible for the observed association.
Then, you find another graph, which shows that the risk of Down syndrome increases with maternal age. There’s no doubt that maternal age is also associated with birth order. Mothers will give birth to their fourth or fifth child are on average older than those who have their first baby. When you look at the risk of down syndrome within each age group, birth order doesn’t seem to matter at all. In summary, maternal age is entirely responsible for the association between two other variables, birth order and Down syndrome. This effect of an extraneous variable that wholly or partially accounts for the apparent effect of the study exposure or that masks in the underlying true association is called confounding. If you hadn’t looked further, you might have thought that birth order might cause Down syndrome, which is clearly not true. Confounding can be a real headache for researchers, and if not properly controlled for, it can produce misleading results.
3.2 How to detect Confounding
Confounding can lead to biased estimates and produce misleading results. Therefore, it is something that we should know about when designing, conducting, or critiquing a study. But how can we know if this confounding? There’s no statistical test for confounding, that are of course statistical methods that can help us make an informed decision. But it depends largely on our judgement. We will look at four commonly used ways to identify potential confounding factors in an epidemiological study.
Let’s consider an example of a study which aims to investigate the association between dog ownership and mortality among the elderly. Some previous studies have found that owning a dog can be associated with higher life expectancy. One straightforward way to identify factors that could confound this association, is to explore the literature. Knowledge of the subject matter can heavily influence our decisions regarding confounding. For example, if other studies have shown evidence that the size of the city where people reside is a confounder in the association between dog ownership and mortality, we have every reason to consider it as a confounder in our study. Knowledge of plausible biological pathways can similarly help us identify confounders.
However, this is not always possible, especially when we explore novel associations for which prior research is scarce. In such cases, we can examine whether the variable of interest satisfies the following three conditions. It is associated with the exposure in the source population, it is associated with the outcome in the absence of the exposure, and it is not a consequence of the exposure. In other words, it is not in the causal path between the exposure and the outcome. If we stick to the same example of dog ownership, our exposure, and mortality, our outcome, and we would like to explore whether age may be a confounder, we would need to answer the following questions. Is age associated with dog ownership among the elderly? Is age associated with mortality among those who do not own a dog? Is aging in the causal path between dog ownership and mortality? We can only respond to the first two questions when we analyze data from the study. But let’s assume that age is associated with both the exposure and the outcome. The answer to the last question is obvious here, owning a dog cannot change your age. So, age is not in the causal path. Age satisfies all three conditions. Therefore, we identify it as a confounder in this study.
A different way to think about this is to stratify data by the variable of interest, which is age in our example, and compare the stratum specific estimates with the estimate that we get when we analyze the entire set of data from the study. In our study, we will need to split our sample by age, below 80 and 80 and above for example, and calculate the odds ratio in each subgroup. We might find that owning a dog reduces mortality by 40 percent among those below 80 years old and 38 percent among those at least 80 years old. But when we analyze the entire sample together, we could find that owning a dog only reduces mortality by five percent, which, of course, doesn’t make sense when you consider the stratum specific numbers. When the pooled estimate is considerably different from what you would expect based on stratum specific estimates, it is very reasonable to think that there is confounding.
Lastly, the fourth way to detect confounding is the following. Let’s say we use a simple logistic regression model to estimate the crude odds ratio that expresses the strength of the association between dog ownership and mortality in our study. When we include age in the regression model, we estimate the adjusted odds ratio, adjusted for age in this case. If the adjusted odds ratio differs from the crude odds ratio by 15 percent or more, this may indicate confounding by age. This number is arbitrary and may not always reflect true confounding. It could be that we introduce confounding by adjusting for an additional variable. This is not the optimal method to identify confounding but can sometimes flag occasions where further investigation is required.
People often assume that they need to use all these methods, however you only need one of the above methods to identify confounding. If you can make a decision based on your knowledge of the subject matter, you don’t need to stratify or explore whether the three conditions are satisfied. In conclusion, there are multiple ways to think about confounding. But at least to some extent, we need to use our judgement to decide which factors may cause confounding. This is a critical decision because it will inform the design and data analysis of our study.
In summary we can detect confounding in the following ways:
- Subject matter knowledge. Factors identified in existing literature or plausible biological pathways can inform your decisions.
- Three criteria for confounding. You need to examine if the suspected extraneous variable satisfies three conditions. – It is associated with the study exposure in the control group (source population) – It is associated with the study outcome in the absence of study exposure – It is not a consequence of exposure, i.e. it is not in the causal path between the exposure and the disease.
- Stratification. Stratify data by the extraneous variable to examine if the estimates within strata of the extraneous variable are similar in magnitude and appreciably different from the crude (pooled) estimate.
- Statistical adjustment. Controlling for the extraneous variable, e.g. by logistic regression, appreciably (>15%) alters the estimate of the association between the exposure and the outcome.
4 Dealing with Confounding
Confounding can be addressed either at the design stage, before data is collected, or at the analysis stage. We will also briefly look at Directed Acyclic Graphs, which is a novel way to detect bias and confounding and control for them.
4.1 Design stage
Confounding can lead to biased estimates which essentially defeats the purpose of research. What is the use of a study if we cannot trust its results? To overcome this problem, we always try to control for confounding. We will look at three methods which you can use to control for confounding at the design stage of a study: randomisation, restriction, and matching.
The first and admittedly the best available method to control for confounding is randomisation. When we split our sample into exposed and non-exposed at random, we ensure that the distribution of all factors and characteristics that may influence the outcome is similar between the two groups. With a large enough sample, this neutralizes the impact of any potential confounding factors. The beauty of randomisation is that it controls, not only for known confounders, but also for those that we are not even aware of. Unfortunately, randomisation only applies to trials. For example, we cannot randomise exposure such as smoking or air pollution due to ethical and practical reasons. Therefore, there are certain questions that cannot be answered by conducting a randomised trial. In such cases, we must rely on other methods to control for confounding.
Restriction is such a method. The idea behind restriction is very simple. We restrict the eligibility criteria for subjects to be included in the sample so that we only study subjects within one category of the confounding variable. For instance, if we think that sex may be a confounder, we can decide to restrict our study to women. This solves the problem of confounding in a simple, efficient, and inexpensive way. On the other hand, it might make recruitment of participants more difficult, and in any case, it undermines the generalizability of the study. Finding that the drug is effective among women does not necessarily mean that it would be equally effective among men.
The third method to control for confounding, which is quite popular for case-control studies, is matching. In matching, we pair one or more controls to each case based on their similarity with regard to selected variables which will consider potential confounders. For instance, we suspect that sex and age maybe confounders in our study. We’ll recruit a case who is a woman aged 54 years. If we conduct a match case-control study, we need to find one or more controls that are 50-year old women. This can increase statistical power in our study, but it requires analytical methods that consider the match design. Also, there’s a limit to the number of confounders that we can control for with matching. If we try to match on too many variables, recruitment of controls becomes impractical. We’re also unable to study the variable we use for matching. Importantly, matching cannot be undone, and matching on a variable that is not a confounder actually harms statistical efficiency. So, a decision to match should be well thought out.
It is not always possible to anticipate and control for confounding at the design stage. Luckily, there are additional methods that can be applied during data analysis.
4.2 Analysis stage
Ideally, when designing a study, you would like to know all the potential confounding factors and plan how to control for them in advance, but some other confounding factors may only be identified as such when data is analyzed. We will look at the two main strategies to control for confounding at the data analysis stage: Stratification and Multi-variable regression models.
So, let’s say we have conducted a study where the exposure is smoking, and the outcome is chronic lung disease. We suspect that age is a confounder in this association. What can we do at the Data Analysis stage? One option would be stratification. The first step is to stratify our data by age group, and obtain an estimate for the association between smoking and chronic lung disease in each stratum. This means that we calculate an odds ratio, for example, for people 20-29 years old, and now the odds ratio for those 30-39 years old, and so on. In the second step of the process, we calculate a weighted average of the stratum-specific odds ratios. This weighted estimate is called Mantel-Haenszel adjusted odds ratio, and this is essentially the results of our study after controlling for confounding by age. This method allows us to get a sense of what is happening within the different strata, but it becomes really burdensome if you try to control for multiple confounders, and it doesn’t really work for confounding variables which are continuous.
A second option, which is what the majority of researchers do nowadays, is statistical adjustment using regression models. In our example, we can estimate the association between smoking and chronic lung disease by fitting a logistic regression model, where the exposure is the independent variable, and the outcome is the dependent variable. If smoking is the only independent variable we include in the model, we will calculate an unadjusted odds ratio. If we wish to control for confounding by age, we simply need to add it as an additional independent variable in the regression model, and we can easily calculate an odds ratio that is adjusted for age. The great advantage of multivariable regression is that we can control for multiple confounding factors at the same time, although including too many variables can sometimes cause problems.
4.3 Directed Acyclic Graphs (DAGs)
There are many other strategies that epidemiologists employ to control for confounding, some more popular than others which include directed acyclic graphs, or simply DAGs, which have become quite popular among researchers in recent years. The directed acyclic graphs, are, as you would expect, graphs. They are essentially a way to present, in a graph, causal associations between variables. If we consider the association between high blood pressure and stroke, High blood pressure is the exposure and stroke is the outcome.
If we think that high blood pressure causes stroke, we will draw an arrow from high blood pressure to stroke. This is a simple way to illustrate what we are talking about. But things are rarely that simple. Let’s introduce one additional factor, age. Old age may affect blood pressure, but it can also affect the probability of having a stroke. To illustrate this, we would add two more arrows in the graph, one going from age to blood pressure and the other also starting from age and going to stroke. And here it is, this is a DAG. Depending on the context of the study, we could add more variables and arrows. Although it becomes quite complicated when you have multiple factors and complex relationships among them.
Using DAGs can help us think about the variables that are relevant to our study and the associations between them. It is also a great tool to communicate this information to others. There’s more to it. Epidemiologists have developed a set of rules called D-separation rules which allow them to identify confounding and other types of bias just by looking at the DAG. One of the benefits of using DAGs is that it is very practical. Applying the D-separation rules, you can identify the minimum set of variables that you need to control for in order to address any sources of bias in your study without having to name or characterize the type of bias that you observed. This is why the focus in DAGs is on confounding and not on confounders.
5 Effect modification
When we analyze data from an epidemiological study, we usually build a statistical model with the aim to describe what has happened in our study. To do so, we make assumptions and often, intentionally ignore differences between individuals or subgroups, so that we can estimate an average association between the exposure and the outcome that applies to the entire population. But sometimes, after controlling for confounding and bias, there is still a third variable, the impact of which on the association between exposure and outcome is so important that cannot and should not be ignored. This is called effect modification.
Imagine you are conducting a randomised clinical trial which aims to test the effectiveness of a new antibiotic against pneumonia. Some of the patients received this new antibiotic, and the rest are given the older drug that is widely used. You follow all the patients up and there are two potential outcomes, a patient can either recover or die. When you analyze data from the entire sample, you find that the odds ratio of recovery of those exposed to the new drug compared to those exposed to the old drug is 1.5, which means those taking the new antibiotic are 50 percent more likely to recover compared to the controls. This is an important result for the trial and if you have conducted your RCT properly, you don’t need to worry about confounding. But before you publish your results, one of your colleagues decides to stratify the data by sex, and notices that the odds ratio is 1.1 for men and 1.9 for women. Men and women do not differ in terms of age, comorbidities, or other confounding factors. After careful consideration, your team decides that the bias cannot explain this difference. So, what’s happening?
Well, sometimes a drug can be more effective in women compared to men, or vice versa. In other words, sex modifies the association between the drug, your exposure, and recovery, your outcome. This is a phenomenon that we call effect modification. Making the definition more general, we say that effect modification exists when the strength of the association varies over different levels of a third variable. In such cases, reporting the overall estimate would not be helpful at all because it would not reflect what actually happened in either sex.
Should you then find a way to control for effect modification and avoid this problem? Definitely not. Unlike confounding, effect modification is a naturally occurring phenomenon. It’s not a problem of your study. You should have no intention to control for it, but the way you report your results should take it into account. In the case of the trial with the new antibiotic, you simply need to present results stratified by sex. You might need one more table in your paper, but this will allow you to accurately report your findings for both men and women. In general, when effect modification is detected, you must conduct stratified analysis. In the example above, we ignored uncertainty. You probably noticed that we gave the estimates without their confidence intervals.
In real life uncertainty cannot be ignored, and this raises one key question, how can we be certain that the stratum-specific estimates are truly different between them? There are statistical methods that can help us identify effect modification such as the Breslow-Day test, the Q test, and including interaction terms in regression models. Regression models are very frequently used, and the term interaction is often considered equivalent to effect modification. The term synergism means that the effect modifier potentiates the effect of the exposure, and antagonism means that the effect modifier diminishes the effect of the exposure. Effect modification is an important concept in epidemiology because it is relevant to many associations in nature but also one that confuses a lot of people. Perhaps it’s because we’re so used to trying to eliminate bias and confounding, that we find it hard to accept that this is a natural phenomenon that we simply need to describe.
6 Confounding vs Effect modifcation
We have discussed how to identify confounding, and separately, how to identify effect modification. But things seem to get a bit confusing when you have to assess both confounding and effect modification in the same study. In reality, there’s absolutely no reason to get confused. In a typical study, we have an exposure and an outcome. Let’s also consider a third extraneous variable. I call it extraneous because it is neither the exposure nor the outcome. It could be something like sex or race, for example. You would like to explore whether the extraneous variable is a source of confounding or effect modification or maybe both.
The first thing to do would be to stratify the data by the extraneous variable, and estimate the association between the exposure and the outcome in each stratum. In practical terms, this means that you obtain an odds ratio for men and one for women, if sex is the extraneous variable of interest, of course. If the odds ratio for men is similar to the odds ratio for women, then based on the definition, there’s obviously no effect modification by sex, while the question whether there is confounding by sex is still open. Using the stratum-specific odds ratios, you can estimate an adjusted odds ratio, adjusted for sex. If the adjusted odds ratio is similar to the crude or unadjusted odds ratio, there is probably no confounding by sex, and you don’t need to take any further action.
But if the adjusted odds ratio differs considerably from the unadjusted estimate, this may be an indication of confounding, and you should control for it by presenting the adjusted estimate. What happens if the stratum-specific estimates are different? In our example, what should you do if the odds ratio for men is statistically different from the odds ratio for women? The answer is straightforward. This is a textbook case of effect modification. Therefore, you will just report the stratum-specific odds ratios separately. Again, the question whether sex is also a confounder has not been answered at this stage. However, if you’re presenting separate estimates for men and women, which you do, because there is effect modification, you don’t really care if sex can cause confounding. In practice, you have already controlled for confounding by presenting stratum-specific odds ratios. This strategy should allow you to identify confounding and effect modification in a study. If you think about it, you already know a few methods to assess confounding and you described one of them, while also repeating the method to identify effect modification. In some cases, you might find that a certain variable is both a confounder and an effect modifier, which is possible. In summary, confounding is a problem of our study, and therefore, we try to control for it. Whereas, effect modification is a natural phenomenon, which requires the presentation of stratum-specific estimates.
7 Causation
The distinction between association and causation is fundamental in epidemiology. Whenever you observe an association between an exposure and an outcome, you have to go through a few alternative explanations before you even start thinking about causality. The first possible explanation is chance which is surprisingly often ignored. There is an entire field of science, statistics, which deals with the uncertainty surrounding research findings. You should always consider the level of uncertainty and how chance may have affected your results.
Let’s assume that you have done all the appropriate statistical tests and you are confident that it is unlikely chance was responsible for your findings. The next step is to think about potential sources of bias and confounding. You’re familiar with the main types of bias and with methods to identify and control for confounding. These must be applied rigorously to exclude any external influences or systematic errors that might have affected your study. Once you’ve concluded that there was no bias or confounding, would you be ready to declare that the association you have detected is causal?
Not exactly. Unless you’re a real pioneer in science, your study is probably not the only one that has investigated this research question. Looking at the bigger picture allows you to make an informed judgement within the context of current scientific knowledge. British epidemiologist, Sir Austin Bradford-Hill, came up with nine criteria that can be helpful in this process. It’s been more than 50 years since he published the list, but I think that there’s still value in considering them. The first thing to consider is the strength of the association. A small effect size doesn’t mean that it’s not causal, but if the association is strong, causality may be more likely. The second criteria in the list is consistency. Consistent findings observed by different people in different settings with different samples can also be an indication of causality.
Causation is also likely in the case of a very specific population at a specific site and disease with no other likely explanation. The more specific and association between a factor and an effect is, the bigger the probability of a causal relationship. Another consideration is temporality. This is an obvious one, the effect has to occur after the cause. It is also helpful if you find that there is a biological gradient in the association. Greater exposure often leads to greater incidence of the disease. Although this is not always the case. Additionally, you can draw evidence from other fields of research. If there is coherence between your epidemiological results and findings from laboratory research and if there is plausible biological explanation of the association, causality becomes more likely. The same is true when you have experimental evidence available and when you consider the effects of similar factors. The Bradford Hill criteria is not a checklist that you need to follow, but they highlight the challenges you might face when thinking about causal inference. Of course, these become relevant only after you have work hard to eliminate factors such as chance, bias and confounding.