Your Systematic Review Probably Has a Methodology Problem. Here’s How to Fix It.
A systematic review is one of the most powerful things you can publish. It also has more ways to go wrong than almost any other study design. Pham et al., writing in Applied Health Economics and Health Policy (2026), mapped the methodology and quality of health economic evaluations in breast cancer screening and found exactly what you’d expect: enormous variation in how reviews report their methods, and significant gaps in quality assessment. This isn’t a niche finding. It reflects a problem that cuts across disciplines.
So if you’re planning a systematic review, or you’re already knee-deep in one and something feels off, this post is for you.
Why Systematic Review Methodology Goes Wrong Early
Most problems in a systematic review don’t appear at the analysis stage. They appear at the design stage, and you only discover them later. The most common culprit is a research question that’s too broad. If your PICO (Population, Intervention, Comparison, Outcome) framework is vague, your search string will be vague, your inclusion criteria will be vague, and your synthesis will be a mess. Therefore, before you run a single database search, write your eligibility criteria out in full and ask a colleague to read them cold. If they can’t tell you which studies would be excluded, your criteria aren’t tight enough yet.
In addition, register your protocol. PROSPERO exists precisely so that reviewers commit to their methodology before they see the results. However, a surprising number of researchers skip this step, either because they don’t know about it or because they’re worried the registration will slow them down. It won’t slow you down meaningfully. What it will do is protect you from unconscious bias creeping into your inclusion decisions, and it will make your eventual paper much harder for reviewers to reject on methodological grounds.
Searching Smarter, Not Just Searching More
Here’s an opinion that might irritate you: searching five databases instead of three does not automatically make your review more rigorous. What matters is whether you’ve searched the right databases for your field and documented your search strings in full. For example, a review in health economics that skips EconLit is missing a significant body of literature. A review in education that doesn’t check ERIC has the same problem. So identify the two or three databases that are genuinely canonical in your area, then add one or two broader ones like Scopus or Web of Science as a net.
Also, save your exact search strings with date stamps. Reviewers will ask for them. Editors will ask for them. You’ll thank yourself when you do.
Quality assessment is where systematic reviews most often get quietly sloppy. Tools like GRADE, AMSTAR-2, or the COSMIN framework (used in measurement property reviews) exist for good reason. However, many researchers pick whichever tool they’ve heard of rather than whichever tool fits their evidence type. As a result, the quality ratings end up meaning very little. Match your appraisal tool to your study designs. If your included studies are economic models, CHEERS is more appropriate than a generic risk-of-bias tool.
Making Your Meta-Analysis Actually Say Something
If you’re running a meta-analysis alongside your review, heterogeneity is the thing that will haunt you. A high I² statistic isn’t a failure. In fact, it’s information. Therefore, don’t just report it and move on. Run subgroup analyses or meta-regression to explore where the variation is coming from. This is where a systematic review becomes genuinely useful to the field rather than just a box-ticking exercise.
Report your effect sizes with confidence intervals, not just p-values. State clearly what a clinically or practically meaningful effect would look like in your field. Readers need that anchor.
And finally, write your limitations section like you mean it. Not a perfunctory paragraph at the end, but a genuine account of what your review can and cannot tell us.
The field doesn’t need more systematic reviews that technically follow PRISMA while quietly cutting corners on methodology. It needs yours to be one of the good ones. So: what’s the weakest part of your current protocol?
Image: Photo by Antony Hyson Seltran on Unsplash
Explore more insights at Research Partners.