As evaluation of health interventions involving machine learning or other AI systems moves into clinical trials, NDORMS researchers from the Centre for Statistics in Medicine (CSM) and the EQUATOR network joined an international group to developed guidelines aiming to improve the quality of these studies and ensure that they are reported transparently.
Gary Collins, Professor of Medical Statistics, NDORMS who was involved in the development of the guidelines said: “Health care professionals and policy-makers need to be confident that an AI intervention is safe and effective before it is used in practice. This new set of guidelines take an important step forward by providing a consensus-based framework to guide the development of study protocols to evaluate AI interventions, as well as guidance on key information to report when publishing the findings of these new AI intervention studies.”
The research was led the University of Birmingham and University Hospitals Birmingham NHS Foundation Trust (UHB) in collaboration with leading institutions from across the world, including Oxford. The findings and the new guidelines have been published in Nature Medicine, The BMJ and The Lancet Digital Health.
Development of the new reporting guidelines, which expand on the current SPIRIT 2013 and CONSORT 2010 reporting frameworks, will boost transparency and robustness for clinical trials evaluating AI health solutions.
SPIRIT-AI extension is a new guideline for clinical trials protocols and CONSORT-AI extension is a new reporting guideline for clinical trial reports, for evaluating interventions with AI components.
Future clinical trials evaluating an AI intervention will be expected - and often required - to report their publications to the new standards. The guidelines will also help medical professionals, regulators, funders and other decision-makers assess the quality of planned clinical trials and assess whether the algorithm is safe and likely to bring about patient benefit.
Professor Alastair Denniston, Lead for AI at Birmingham Health Partners Centre for Regulatory Science and Innovation, and Consultant Ophthalmologist at UHB, commented: “Patients could benefit hugely from the use of AI in medical settings, but before we introduce these technologies into everyday practice we need to know that they have been robustly evaluated and proven to be effective and safe. Our previous work showed just how big a problem this can be and that we needed a way to cut through the hype surrounding AI in healthcare.
“These new reporting guidelines – SPIRIT-AI and CONSORT-AI – provide a solution to the ‘hype’ problem. They provide a clear, transparent framework to support the design and reporting of AI trials that will help to improve quality and transparency. These extended guidelines will help to reduce wasted effort and deliver effective AI-led medical interventions to patients quicker.”
The new guidelines are the first in a series that will help shape the reporting of AI research. Professor Collins is leading the development of TRIPOD-AI, for studies developing and evaluating AI diagnostic and prognostic algorithms, as well as being involved in the STARD-AI guideline for developing guidance on AI diagnostic accuracy studies.