A paper can be technically sound and still answer a narrower question than its headline suggests. My first pass is therefore less about deciding whether I agree and more about reconstructing the question the study was designed to answer.
1. Who was studied?
I look beyond the condition name. Inclusion and exclusion criteria reveal the population: severity, treatment history, age range, comorbidities, setting, and other characteristics. I then ask how closely that group resembles the people to whom the conclusion is being applied.
2. What exactly was compared?
“Treatment” is not enough. I note the dose, schedule, duration, formulation, and permitted co-interventions. I examine the comparator with equal care. Placebo, usual care, and active treatment each produce a different question, and an inadequate comparator can make an intervention look more useful than it would in practice.
3. Which outcome carried the study?
The primary outcome should be prespecified, clearly defined, measured at a meaningful time, and analysed in a way aligned with the research question. Secondary outcomes can add context, but multiple analyses increase the chance of impressive-looking findings that are less secure. I also distinguish patient-important outcomes from biomarkers or other surrogates.
4. How large and how precise was the effect?
A p-value does not tell me the size or practical importance of a difference. I look for the effect estimate, absolute as well as relative measures where possible, and its confidence interval. A wide interval may be compatible with meaningfully different conclusions. ICH E9 recommends accompanying treatment-effect estimates with confidence intervals whenever possible.
5. What might change my interpretation?
Here I consider missing data, departures from assigned treatment, loss to follow-up, selective reporting, funding and conflicts, protocol changes, and whether the safety observation period was adequate. I compare the paper with its protocol or registration record when available. I also ask whether the limitations described by the authors are the limitations most likely to matter.
A useful reading is not “positive” or “negative.” It is a bounded statement: in this population, under these conditions, this is what the study suggests—and this is how uncertain that suggestion remains.
The final sentence test
Before I move on, I write one sentence with five parts: population, intervention, comparator, outcome, and uncertainty. If I cannot do that, I probably do not understand the paper yet. If my sentence is much more cautious than the headline, that difference is itself important information.