Statistical Thinking in the Age of AI for Drug Development
Zhiheng Xu (FDA)
Highlights:
The field of statistics has undergone significant transformations along with the rapid progress AI has made in the last few decades.
While AI holds potential for advancing drug development, it cannot replace statistical thinking beyond the data— trial design, research questions, model checking, bias mitigation, etc.
Statistical thinking underlies the successful implementation of all three domains representing the evolution of statistical methodologies in drug development: statistical inference, statistical modeling and statistical learning.
Practice statistical thinking in solving real-world problems in areas including but not limited to biomarker, disease assessment, de-centralized clinical trials (DCT), real-world data (RWD), and complex innovative trial design (CID)
It was a sizzling summer in 1956 in Hanover, New Hampshire where the Dartmouth Summer Research Project on Artificial Intelligence hosted by John McCarthy and Marvin Minsky was taking place [1]. In this historic conference, McCarthy, imagining a great collaborative effort, brought together top researchers from various fields for an open-ended discussion on artificial intelligence, the term which he coined at that very event with the aim of discussing thinking machines. That was when the term artificial intelligence (AI) was first introduced. Figure 1 displays the Dartmouth Hall where the 1956 Dartmouth AI workshop took place.
Figure 1: Dartmouth Hall, where the 1956 AI workshop took place (Image from James Moor [1]).
The research on AI took off over the next two decades following the 1956 workshop as computers became faster, cheaper, and could store more information. Early progress was made in problem-solving and the interpretation of spoken language, respectively [2]. However, as AI progressed, the computers that sped up the AI development were holding AI back because of the lack of computational power to do anything substantial. The computers weren’t fast enough and couldn’t store enough information. AI research stagnated, a period later referred to as the “AI Winter” of the 70s [3]. There were short AI booms in the 80s but these were met with a second AI winter in the late 80s and early 90s [3]. In the late 90s and onward, AI achieved many significant milestones, including defeating chess Grand Master Gary Kasparov in 1997 with IBM’s Deep Blue and defeating GO champions in 2015 and 2016 with Google DeepMind’s AlphaGo [4]. In the first decades of the 21st century, AI development was fueled by interrelated innovations: access to large amounts of data (known as “big data”), cheaper and faster computers, and the implementation of advanced machine learning techniques such as deep learning. Subsequently, the large language model (LLM) came along with the debut of ChatGPT in late 2022 which has triggered widespread public interest and discussion on AI [5]. Figure 2 displays the history of AI since the 1956 Dartmouth AI workshop.
Figure 2: The history of AI
AI has impacted many fields, and drug development is no exception. For example, Liu and others [6] conducted an analysis of regulatory submissions of drug and biological products to the FDA from 2016 to 2021 and found an increasing number of submissions that included artificial intelligence/machine learning (AI/ML). Realizing the potential of AI/ML in drug development and public interest, FDA CDER released two documents related to AI/ML in drug development in 2023 in which the Agency provided a high-level overview of the diverse and evolving uses of AI/ML being employed throughout the drug development process [7, 8].
As AI has progressed rapidly in the last few decades, the field of statistics has also undergone significant transformations. Frank Bretz and Joel Greenhouse outline three domains (statistical inference, statistical modeling, and statistical learning) which represent the evolution of statistical methodologies in drug development [9], as shown in Figure 3. Although Bretz and Greenhouse depict the evolution of statistical methodologies in a chronological order, this doesn’t mean that any of the three domains are outdated or inferior to the others. In drug development, all three domains are essential and supplement each other in addressing the emerging challenges of analyzing new data and validating innovative technologies. Statistical thinking underlies the successful implementation of all three domains.
Figure 3: The evolution of statistical methodologies in drug development (Image from Bretz and Greenhouse [9])
There are usually four steps in the process of statistical thinking (Figure 4): developing a research question, data collection, statistical analysis, and reporting study findings. Throughout this process, statistical thinking should be combined with thinking from the computational, scientific, ethical and human perspectives [10].
Figure 4: The process of statistical thinking
How can statisticians put our statistical thinking in practice? In the talk I gave at ASA Boston Chapter on February 23, 2024, I used five examples (biomarker, disease assessment, de-centralized clinical trials (DCT), real-world data (RWD), and complex innovative trial design (CID)) to demonstrate how we can utilize the statistical thinking in solving real-world problems. These are important areas of drug development in which AI might play a significant role. For example, in disease assessment, there is an ongoing debate between the use of blinded independent central reviewer (BICR) and local evaluator (LE) in assessing the status of disease progression from radiological images. One of the trending discussion points is how to use AI-assisted algorithm in such disease assessments. Among the increasing number of AI/ML devices approved by FDA, radiology is the primary area that has seen the wide adoption of AI/ML technology [11]. There are some advantages in using AI-assisted algorithms in disease assessment, such as greater clarity, better standardization, and improved decision traceability. However, there are still some challenges in using AI. Pennello and Samuelson [12] listed many statistical challenges in validating those AI-based medical devices, including issues in the interpretability, uncertainty, reproducibility, confounding, representativeness, etc. Xu and others also discussed challenges and opportunities in areas such as data quality control, bias issue and mitigate strategies when applying AI/ML in precision medicine [13].
As new data types emerge, such as digital health technology (DHT) and real-world data (RWD), and new designs such as CID and DCT come into the drug development, a critical question is how we can build confidence in these new data and new methods. For example, when considering using RWD as an external control, AI and ML offer the computational power and analytical sophistication needed to check data comparability, accuracy, reproducibility, traceability, etc. In February 2023, the FDA released a draft guidance on externally controlled trials for drugs and biological products which includes a checklist for assessing data comparability [14]. Another key question is whether such methods and data can be treated as "fit-for-purpose" and for what purpose they might be fit. For example, a clinical outcome assessment (COA) is considered fit-for-purpose when the level of validation associated with a medical product development tool is sufficient to support its context of use [15]. Another example is the Bayesian Optimal Interval (BOIN) design, which was determined fit-for-purpose for phase I dose finding clinical trials [16]. The FDA’s Fit-for-Purpose (FFP) Initiative provides a pathway for regulatory acceptance of dynamic tools for use in drug development programs [16].
While AI holds potential for advancing drug development, it cannot replace statistical thinking beyond the data— trial design, research questions, model checking, bias mitigation, and so on. Statisticians should embrace AI, and in response, the field of AI will benefit from increased statistical thinking.
Disclaimer: the paper reflects the views of the author and should not be construed to represent FDA’s views or policies.
References
1. Moor, James. The Dartmouth College Artificial Intelligence Conference: The Next Fifty Years. AI Magazine, 27(4), 87, 2006. https://doi.org/10.1609/aimag.v27i4.1911
2. The history of AI. Accessed at https://en.wikipedia.org/wiki/History_of_artificial_intelligence on Feb. 23, 2024.
3. AI winter. Accessed at https://en.wikipedia.org/wiki/AI_winter on April 18, 2024.
4. Gary Kasparov. Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins. PublicAffairs; 1st edition (May 2, 2017).
5. OpenAI. Accessed at https://www.openai.com on April 19, 2024.
6. Qi, Liu, et.al. Landscape Analysis of the Application of Artificial Intelligence and Machine Learning in Regulatory Submissions for Drug Development From 2016 to 2021. Clinical Pharmacology & Therapeutics, 113: 4: 771-774, 2023.
7. AI in Drug Manufacturing Discussion Paper (2023). Accessed at https://www.fda.gov/science-research/science-and-research-special-topics/artificial-intelligence-and-machine-learning-aiml-drug-development on Feb. 23, 2024.
8. AI/ML for Drug Development Discussion Paper (2023). Accessed at https://www.fda.gov/science-research/science-and-research-special-topics/artificial-intelligence-and-machine-learning-aiml-drug-development on Feb. 23, 2024.
9. Frank Bretz & Joel B. Greenhouse (2023) The Role of Statistical Thinking in Biopharmaceutical Research, Statistics in Biopharmaceutical Research, 15:3, 458-467, DOI: 10.1080/19466315.2023.2224259
10. Baillie, M., Moloney, C., Mueller, C.P., Dorn, J., Branson, J., and Ohlssen, D. Good Data Science Practice: Moving toward a Code of Practice for Drug Development. Statistics in Biopharmaceutical Research, 15, 74–85, 2022.
11. AI/ML-Enabled Medical Devices. Accessed at https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-aiml-enabled-medical-devices on April 19, 2024.
12. Gene Pennello, Frank Samuelson. AI-Enabled Medical Devices and Diagnostics: Statistical Challenges and Opportunities from a Regulatory Perspective. AMSTAT News, September 1, 2023.
13. Zhiheng Xu, Bipasa Biswas, Lin Li, Billy Amzal. AI/ML in Precision Medicine: A Look Beyond the Hype. Therapeutic Innovation & Regulatory Science. May 2023. https://doi.org/10.1007/s43441-023-00541-1
14. Considerations for the Design and Conduct of Externally Controlled Trials for Drug and Biological Products. FDA Draft Guidance. Accessed at https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-design-and-conduct-externally-controlled-trials-drug-and-biological-products on Feb. 23, 2024.
15. Patient-Focused Drug Development: Selecting, Developing, or Modifying Fit-for-Purpose Clinical Outcome Assessments. FDA Draft Guidance. Accessed at https://www.fda.gov/media/159516/download on Feb. 23, 2024.
16. Drug Development Tools: Fit-for-Purpose Initiative. Accessed at https://www.fda.gov/drugs/development-approval-process-drugs/drug-development-tools-fit-purpose-initiative on April 19, 2024.