Who doesn't want to know if drinking a second or third cup of coffee a day will improve your memory, or if you sleep too much, it increases your risk of heart attack?
We are invested in staying healthy, and many of us are interested in reading about new research findings to help us understand our lifestyle choices.
But not all research is equal and not every research finding should be interpreted the same way. Nor do all media headlines reflect what was actually studied or found.
So how can you say? Keep these five questions in mind when reading media stories for new studies.
1. Was the research reviewed?
Peer review is a process by which the study is reviewed by discipline experts to assess the scientific validity of the study.
This process involves a researcher writing his methods and results of the study and sending it to a journal. The manuscript is then usually sent to two to three peer review experts.
If there are major shortcomings in a study, it is either rejected for publication, or the researchers are made to address these shortcomings.
Although the review process is not perfect, it does show that the study has been reviewed.
Peer review has some problems – but the scientific community is working on it
Any reported findings for which no reviews have been reviewed should be read with some degree of reservation.
2. Was the study conducted in humans?
Findings from animal studies such as mice or cells in the laboratory (also called in vitro studies) represent the earliest phase of the scientific discovery process.
No matter how interesting they may be, hard claims about human health should not be made solely on the basis of these types of studies. There is no guarantee that the findings from animal or cell studies will ever be replicated in humans.
3. Are the findings likely to be a cause-and-effect relationship?
For a study to be relevant to our daily health, the findings should reflect a causal relationship instead of just correlation.
If one study has shown that drinking coffee is linked to heart disease, for example, we want to know if it's because coffee is actually coffee. caused heart disease or whether these things happened together.
In a number of studies that found this association, researchers subsequently found that coffee drinkers were probably smokers and therefore, these results were more likely to reflect a genuine causal link between smoking and heart disease.
In inference studies, where researchers observe differences in groups of people, it can sometimes be difficult to separate the relationship between variables.
The highest degree of evidence regarding causality comes from double-blind placebo-controlled randomized controlled trials (RCTs). This experimental type of study, where people are divided into groups to randomly receive an intervention or a placebo (treatment embarrassment), is the best way we can determine if something is causing the disease. However, it is also not perfect.
Although other types of human studies play an important role in our understanding of health and disease, they can only highlight associations that are not indicative of causal relationships.
Clearing confusion between the correlation and the cause
4. What is the magnitude of the effect?
It is not enough to know that exposure (such a third cup of coffee or more than nine hours of sleep a night) causes an outcome, it is also important to clearly understand the strength of this relationship. In other words, how much is the risk of the disease increased if you are exposed?
If it is reported that the risk of the disease increases by 50% (which is a relative risk), this sounds pretty daunting. However, if the original risk of the disease is low, then a 50% increase in risk may not represent a very real increased risk of the disease. An increased risk of a disease of 50% can mean a 0.1% risk of a disease risk of 0.15%, which does not sound so dramatic.
What you need to know to understand risk assessments
5. Is the finding confirmed by other studies?
A single study itself, even if a well-conducted randomized controlled trial, can never be considered definitive evidence of a causal link between exposure and disease.
Because people are complex and there are so many variables in each study, we cannot be sure that we understand what is happening until the findings are replicated in very different groups of people, using very different approaches.
Until substantial evidence is found to be consistent, we must be very careful about our interpretation of the findings of any study.
What if these questions are not answered?
If the media reports that you are reading, do not answer these questions, consider switching to news websites or reviewing the original work. Ideally, this will be linked to in the news article you are reading, or you can search PubMed for the article using a few keywords.
The abstract of the journal article should tell you about the type of study, whether it was conducted on people or the size of the effect. If you are not blocked by paywall, you may be able to see the full article in the journal that should answer all the questions you have about the study.
Where's the evidence in science? Nothing