As a healthcare professional, it’s a good practice to stay apprised of the latest medical research, particularly as it relates to your discipline. Unfortunately, not every set of findings are as trustworthy or accurate as their publishers would have you believe.
At a time when attention spans are short and time is limited, it’s become common practice for publications to garner attention with flashy, top-line findings. However, a deeper look can reveal red flags that bring these publications’ claims into question. So how can you tell if what you’re reading is too good to be true?
Is it fraud, or simply bad research?
Not every study that arrives at a misleading conclusion does so willfully or maliciously. Despite our preconceptions of scientists, doctors, and researchers, mistakes can be made and perhaps shortcuts taken. Even researchers with the best intentions can sometimes come to conclusions not shared by their peers.
One study, “Reanalyses of randomized clinical trial data” found that of the 37 eligible reanalysis of published clinical trials, “13 reanalyses (35%) led to interpretations different from that of the original article, three (8%) showing that different patients should be treated; one (3%), that fewer patients should be treated; and nine (24%), that more patients should be treated.”
While sloppy research that leads to questionable findings certainly doesn’t help the profession, fraudulent studies are a serious issue in the field of clinical research.
Beware of misleading studies.
The academic and scientific communities are currently contending with “paper mills”: organizations that sell falsified studies and papers to scientists and researchers to publish under their own names, with the aim of meeting the publication goals set forth by their employer. This article from Nature lays out how prevalent the issue is, and how much it affects the research publication industry.
Other studies intend to mislead readers about their results rather than their provenance. Sometimes it may not be possible at first glance to determine that the study you’re looking at is intentionally inaccurate, and that can be by design. One challenge in detecting fraudulent or misleading studies is if the study in question has been manipulated to look legitimate. Researchers may choose to omit data or findings that don’t support the desired conclusion, or may opt to actively alter or fabricate data to achieve the preferred outcome.
All of these complicating factors make it difficult to trust the abstract results you see for a clinical study—which can be a good thing. A healthy degree of skepticism is useful in examining studies, and the best research is conducted, documented, and designed to withstand any such scrutiny. Here are some things to keep an eye out for that can help you to separate the legitimate studies from the dubious ones.
What are some red flags with clinical studies?
The study’s sample size is too small.
This rule makes sense, even for those of us who may not have majored in math: the more often a particular result is achieved, the more likely people believe that that result isn’t random or statistical noise. If a research group is looking to achieve a particular result, it’s much easier to do so within a smaller group of subjects. Think of it this way: if you’re looking to flip a coin and have it land only on heads, you’re better off flipping it twice rather than a dozen times. Even if the small sample is simply down to something as innocent as an inability to field more subjects, it still brings the replicability of the findings into question.
The results presented are unrealistic.
Scientific research is occasionally capable of remarkable breakthroughs, but those tend to be the exception. Typically, the studies we come across in our reading offer smaller—but still significant—advances in our knowledge. More importantly, they tend to be measured in what they claim. As noted in this study by Glenn T.Clark, DDS, MS, and Roseann Mulligan, DDS, MS, a “single study never proves that a hypothesis is true; it can only reject the null hypothesis. While most people are not comfortable using such cautionary language, this is the correct scientific language.”
It’s perhaps not surprising that in an attention economy, some scientists and researchers might want to gain notice in the hopes of advancing their profile and career, especially considering studies that are non-replicable are cited more than replicable studies, and that few of the publication that cite these non-replicable studies ever make note of that issue. This should give most people even more reason to question what’s beyond the splashy headlines.
All or parts of the study are plagiarized.
One of the reddest of flags for any study would be if much of the supposedly original work was copied from elsewhere. We’re all familiar with the most blatant form of plagiarism from our time in school: taking things written by others and presenting them as our own. In this case, plagiarism can also include copying data collected and presented by another study as original and unique. More subtle but equally harmful in the scientific and medical community is using the ideas of another study and presenting them without citation as your own.
Discovering that a study’s author(s) have plagiarized all or parts of a study is an instant way to discredit a study, even if the underlying ideas or data taken from elsewhere are sound.
How to avoid bogus clinical studies.
Given how much uncertainty there may be around any study given the factors listed above, how can you know what to trust when it comes to studies you may see on social media or elsewhere? While there’s no surefire way to avoid bad studies entirely (given that even reputable journals can be fooled), you can take measures to avoid getting duped.
Find sources you can trust.
Most media outlets make a good faith effort to inform readers with the best information, but much of the medical news you’ll read from mainstream outlets isn’t from doctors, and any reporter without a medical background could just as easily fall prey to the trap of spreading the headlines of the attention-grabbing clinical study with dubious design or questionable methods.
One easy way to hopefully find good information is to look to the source to see if the study in question has been peer-reviewed. It’s also worth checking where you’re reading the study from; medical schools and respected medical and governmental bodies are more likely to publish well-vetted studies than websites you’ve never previously encountered. Some well-respected publications in the PT field include:
- the American Journal of Sports Medicine and
- the Journal of Orthopaedic & Sports Physical Therapy (JOSPT).
The American Journal of Medicine and The Lancet are also highly regarded journals for more generalized medical practice.
Understanding the studies you’re reading.
You may be well acquainted with clinical studies, but if not, it’s worth learning how they work and what they are to get a better sense of how accurate you can assume the results to be. Understanding the relationship between the phase of a study and the size of a trial group, who is included and excluded from that group, and the design of the study itself throughout its phases can help to inform the ultimate results with a little more context than just the topline findings. And reading beyond those headlines can help you get at the actual information presented to spot any questionable methods or findings.
Of course, there’s a long way to go from clinical studies to putting treatments into practice, and nothing you read is likely to have an immediate impact on your therapeutic work. But knowing what to look out for and what is worth paying actual attention to, as well as reading past the headlines and into the substance of the study, is helpful in talking with patients about what they might be seeing on the news and giving them the information they need.
