Industry ties and evidence in public comments on the FDA framework for modifications to artificial intelligence/machine learning-based medical devices: a cross sectional study

AbstractObjectivesTo determine the extent and disclosure of financial ties to industry and use of scientific evidence in comments on a US Food and Drug Administration (FDA) regulatory framework for modifications to artificial intelligence/machine Learning (AI/ML)-based software as a medical device (SaMD).DesignCross-sectional study.SettingWe searched all publicly available comments on the FDA “Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning (AI/ML)-Based Software as a Medical Device (SaMD) - Discussion Paper and Request for Feedback” from April 2nd 2019 to August 8th 2019.Main outcome measuresThe proportion of articles submitted by parties with financial ties to industry, disclosing those ties, citing scientific articles, citing systematic reviews and meta-analyses, and using a systematic process to identify relevant literature.ResultsWe analysed 125 comments submitted on the proposed framework. 79 (63%) comments came from parties with financial ties; for 36 (29%) comments it was not clear and the absence of financial ties could only be confirmed for 10 (8%) comments. No financial ties were disclosed in any of the comments that were not from industry submitters. The vast majority of submitted comments (86%) did not cite any scientific literature, just 4% cited a systematic review or meta-analysis, and no comments indicated that a systematic process was used to identify relevant literature.ConclusionsFinancial ties to industry were common and undisclosed and scientific evidence, including systematic reviews and meta-analyses, were rarely cited. To ensure regulatory frameworks best serve patient interests, the FDA should mandate disclosure of potential conflicts of interest (including financial ties), in comments, encourage the use of scientific evidence and encourage engagement from non-conflicted parties.Strengths and limitations of this study-We analysed the extent of financial ties to industry and the use of scientific evidence in comments on the proposed FDA framework-We used a comprehensive strategy to attempt to identify financial ties to industry-Readers may be able to contribute higher quality comments to subsequent drafts of this framework-There is heterogeneity in the degree of conflict with respect to the framework that the recorded financial ties may represent; some ties will be more likely than others to result in biased commenting-Because the framework could not be classified as pro-industry or not, we did not classify the direction of opinions expressed in comments with respect to the framework and their association with financial ties-We do not know how information submitted to FDA is used internally in the rule-making process