Trials on trial

What is a pilot trial and why do we do them?

A Randomised Controlled Trial (RCT) is a study where patients are randomly assigned (randomised) to two groups. All the patients in one group receive a new treatment, the intervention group, and all the patients in the second group receive an existing or no treatment, the control group. These trials help researchers make decisions about which treatments work better and should be given to patients. However, they often take many years to do and can be very expensive, so it is important that when we do them, we do them well.

If researchers are unsure about whether their randomised trial will work, they can opt to do a pilot trial first.

Pilot trials are small versions of the randomised trial we want to do. They let researchers “rehearse” all or part of their randomised trial on a smaller scale before going ahead with it so we can be more confident that it will be a success(1).

What do pilot trials tell us?

When researchers plan their pilot trial, they set targets in advance to help them decide whether to do the future randomised trial or not. We call these targets “progression criteria”. These progression criteria can tell us how to proceed, for example a STOP-GO or a STOP-AMEND-GO (also known as a traffic light) approach.

1. STOP-GO approach:

E.g. These criteria for a pilot trial check if we can recruit and retain enough patients while minimising missing data in order to help us to decide whether we should do a larger randomised trial.

2. STOP-AMEND-GO approach:

E.g. In this pilot trial the same criteria as the trial above are used but there is an additional set of intermediate targets where we can also see whether changes to the trial could be made to improve its design before moving on to a larger randomised trial.

Why we did a review

There is currently no guidance for how to decide what progression criteria to use for these types of pilot trials. We wanted to understand what progression criteria are used by existing pilot trials, how they were developed, and how they helped researchers make decisions about which larger trials (RCTs) they should do in the future.

What we did

We looked at the characteristics of progression criteria that were reported in 160 pilot trial research publications (123 pilot trial protocols and 37 pilot trial results papers). These included what the progression criteria were, what they were based on and who had decided on them. We included publications that were published between January 2018 and December 2019 from four key journals. Our methods are described in detail in a preregistration on the Open Science Framework(2).

Our findings

We found that progression criteria used varied between different pilot trials, but most often included a measure of recruitment (“can we get patients into the trial?”) and retention (“do patients stay in the trial to the end?”).

We found that researchers often did not report how they developed their progression criteria. For example, out of 160 publications only twelve (7.5%) reported who had been involved in developing progression criteria and less than a third of the researchers reported their reasons behind their choice of progression criteria (44/160, 28%).

We found that for many pilot trials researchers reported that they had done, or planned to do, qualitative research (108/160, 68%), for example interviews with patients who are in the pilot trial to find out about their experiences. However, only 31% of the publications that included qualitative research then reported how their qualitative findings would contribute to progression criteria (34/108, 31%).

We found that most of the completed pilot trials reported the intention to proceed to a larger randomised trial (30/37, 81%), even though less than half had strictly met all their progression criteria (17/37, 46%) demonstrating a flexible approach to decision-making based on progression criteria targets.

Finally, we found that many peer reviewers (researchers who review research papers that have been submitted to a journal to help decide if they should be published) commented on progression criteria, with peer reviewers of 35 publications commenting that progression criteria appeared not to be included in the original publication that was submitted to the journal.

A full report of our findings are published Open Access in BMJ Open(3).

What does this mean for future pilot trials?

It was often unclear how progression criteria were established, how they were or will be assessed, and who was involved because this detail was not well reported. Our findings suggest that guidance is needed to improve how researchers report progression criteria for their pilot trials. In the meantime, we suggest researchers consider reporting how they developed their progression criteria (including who was involved and what they are based on) in pilot trial protocols, and report how their progression criteria informed the decision of whether to do a future trial in their pilot trial results publication. We hope that better reporting of pilot trials will help avoid research waste – if we are better prepared to make decisions about which Randomised Controlled Trials(RCTs) are likely to be a success, this should lead to better trials and improved outcomes for patients.

References:

1. Eldridge SM, Lancaster GA, Campbell MJ, et al. Defining feasibility and pilot studies in preparation for randomised controlled trials: development of a conceptual framework. PLoS One 2016;11:e0150205. doi:10.1371/journal.pone.0150205

2. Mellor K, & Hopewell S. An investigation of the current use of progression criteria in external randomised pilot studies: a systematic review protocol. OSF 2020. doi:10.17605/OSF.IO/BN35K

3. Mellor K, Eddy S, Peckham N, et al. Progression from external pilot to definitive randomised controlled trial: a methodological review of progression criteria reporting. BMJ Open 2021;11:e048178. doi:10.1136/bmjopen-2020-048178

Authors of this abstract:

Katie Mellor

Pradeep Virdee

Members of the research team who are based in NDORMS:

Katie Mellor

Nicholas Peckham

Sally Hopewell 

Susan Dutton