Updating systematic reviews and meta-analyses, the easy way

Systematic reviews and meta-analyses are valuable evidence reports that get outdated when new studies become available. Keeping track of the literature is warranted but time-consuming. There is an alternative that works well for many updates.

What are the chances that researchers of a new study don’t cite the original review or any of the studies included? It’s possible, but it’s reasonable to expect that the better studies cite relevant previous studies. It’s just a lot of work to track all citations to all articles. Until now.

CoCites extracts the citations and reference lists of many articles at once and is therewith ideal for updating systematic reviews and meta-analyses. It’s as easy as a few steps and several mouse clicks. The search requires free registration. Here is how it works.

The steps

1. Identify the systematic review.

On your My CoCites page search for the review and add it to a new query set. Go back to the My CoCites page to find the new query set. The My CoCites page and other advanced features are available after (free) registration on the CoCites website.

2. Select the articles from the review.

Click the arrow under the title to show its references. Select all relevant articles and add them to the same query set. You can also select and add them all and then remove irrelevant articles (see recommendations below).

3. Select the search options.

If the number of articles in the query set is small, say, lower than 20, and don’t have many citations, then opt to show all articles, including those that only cite one article from the query set. If you are only interested in newer studies, then you can filter the search results to exclude articles published before the review. Leave unchecked if you want CoCites to find older articles too.

4. Run the search.

It was that simple. You now have a list of articles that cite the articles in the query set, ranked in descending order of the number of citations. At the top of the list you find articles that cite many articles from the query set. You will find that their topics are similar to the review.

The example results

The example in the pictures is taken from a 2008 review about the mental health of incarcerated adolescents. The review was recently updated and we aimed to find the added articles by a citation search on the original review.

The original review included 25 studies. The update added 22 others, 20 of which were published after the original review. We performed a citation search to find articles published since 2008 that cited two or more articles from a query set that included the original review and its key references relevant to the topic, 40 articles in total.

The citation search returned 337 articles. Among them were 14 of the 20 new studies. Of the six ‘missed’ studies, one was a published report and two were not available through PubMed. Thus, the search retrieved 14 of the 17 ‘retrievable’ studies. The remaining three studies, reporting data from Iran, Russia, and Brazil, were available in PubMed, but the authors had not cited two or more articles from the query set. One study appeared in the search results when we re-ran the citation search with the newly retrieved articles added to the query set.

Tips for better searches

In keyword searches, the relevance and completeness of the results depend on the choice of keywords. The quality of citation-based searches depends on the choice of the articles in the query set.

The query set should include articles that are likely cited by the researchers of the newer studies. Suitable articles are the original review, its references, and, for example, articles about studies that were excluded for methodological reasons and other reviews. One of our ‘missed’ studies cited another systematic review on the same topic, published in 2010. The researchers did cite a review, just not the one we were updating.

The query set should be sufficiently large. From our validation study of CoCites, we learned that the citation search worked better if the set included over 20 articles. More is better but …

… the query set should also be specific if the goal is to find similar articles on niche topics. In the example above, we removed highly-cited references (>100s citations) from the query set. These highly-cited articles — such as diagnostic criteria, reviews, and perspectives — are cited for many reasons, and they may flood the search results with articles that are not relevant for the niche topic when a query set includes several of them.

If the update is for an older review, like in our example, keep in mind that researchers may cite recent studies on the same topic but not the older ones. If this is the case, repeat the citation search with the newly retrieved studies added to the query set. You may also consider adding relevant top-ranked articles from the search results, even if these do not meet the selection criteria for the review.

If you use CoCites to update your review, use it besides other strategies to find out if you can rely on the results of the citation search. This may not be the case for every topic and field. CoCites can only find articles that are in NIH’s iCite/PubMed database. The database covers health and medical research from a (wide) curated selection of scientific journals.

How to get CoCites

The option of query sets is available in the full version of CoCites that requires free registration. Register to receive announcements of additional features or follow me, @cecilejanssens, or @cocites on Twitter. Instructions for CoCites’ other features, including the co-citation search, are coming soon.

Professor of epidemiology | Emory University, Atlanta USA | Writes about (genetic) prediction, critical thinking, evidence, and lack thereof.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store