Does peer review consider the search used in systematic reviews?

We are investigating whether peer reviewers or journal editors commented on or raised issues with the literature search during the peer review and editorial decision-making process for systematic reviews submitted to the BMJ and BMJ Open journals between 2020-2021.

What is systematic search?

A systematic review is a type of research study that aims to assess everything already published on a particular topic. It aims to summarise all of this evidence, to see whether there is a clear answer to a particular research question or whether more research is needed.

The first step of a systematic review is to search for every published research study that is relevant to the topic. If the search is not reliable and doesn't find all of the relevant literature, then the review's conclusions cannot be trusted.

To find all of the relevant literature, researchers develop lists of key search terms and all possible variations. For example, they would search for ‘womb’, ‘uterus’, and ‘uterine’, not just one of these related terms. They would also search for all possible spellings, such as 'tumor' and 'tumour'. They also identify the databases and other sources that they should search in. They have to test their search plan properly to make sure it does not miss anything. The researcher developing the search is usually a specialist librarian who has been trained in this area.

What is peer review?

Like other research, systematic reviews are published in academic journals so that their results can be read and used by other researchers, clinicians, patients, and others. Before publishing, journals ask other researchers to read and critically review the articles they are sent, to check the research was done properly and is clearly described. This process is called peer review, and it should make sure that all published research has a certain level of quality.

What is the problem?

Most peer reviewers are not specialists in literature searching. They often don’t have enough knowledge to spot problems in how the search was done or described.

Journals tell their peer reviewers what to focus on when reviewing. However, from what we have seen, most do not provide enough detail on how to review searches.

What will our study do to help?

We will look at two journals and see whether peer reviewers or editors discuss the literature search details of systematic reviews when doing peer review or when making decisions about articles. We will also see whether the peer reviewers and editors make sensible comments about the search.

We will look at the submitted manuscripts and related documents and the peer review reports and editorial decision letters for each systematic review submitted in 2020 and 2021. We will compare these documents with an independent review of the literature search by a specialist librarian.

We hope that the results of this research will help to contribute to current debates about the scope and role of peer and editorial review in the publication process. 

Who we are

This study is led by Shona Kirtley, an information specialist in the UK EQUATOR Centre team. We are collaborating with the BMJ Publishing Group and librarians from the Bodleian Health Care Libraries.