2024 ASA Biopharmaceutical Section Regulatory-Industry Statistics Workshop Session on “Bayesian shrinkage estimation for subgroups: Is it ready for prime time?”
Björn Bornkamp (Novartis), David Ohlssen(Novartis), Mark Rothmann (FDA), Chenguang Wang and Dong Xi (Gilead)
Introduction
On 26th September 2024 a session on Bayesian shrinkage estimation for subgroups took place at the ASA Biopharmaceutical Section Regulatory-Industry Statistics Workshop in Rockville, MD. The session was co-organized by Björn Bornkamp and David Ohlssen from Novartis and was chaired by Dong Xi (Gilead). There were two presentations by Mark Rothmann (FDA) and David Ohlssen, followed by a discussion by Chenguang Wang (Regeneron). The session abstract is provided in the Appendix. This note provides a summary of this session.
Talk 1: Mark Rothmann (FDA/CDER/OTS/OB): “Practical experiences with Bayesian subgroup shrinkage methods for drug trials snapshots”
Bayesian shrinkage estimation for subgroup analysis is ready for primetime. In 2019, the FDA posted on their internet site (https://www.fda.gov/drugs/science-research-drugs/impact-story-using-innovative-statistical-approaches-provide-most-reliable-treatment-outcomes) an impact story on “Using innovative statistical approaches to provide the most reliable treatment outcomes information to patients and clinicians”, which discusses using Bayesian hierarchical models for shrinkage estimation in subgroup analysis, in particular in drug trials snapshots (DTS) [1]. Starting in September 2019, sixteen DTS to date have included subgroup analyses incorporating shrinkage estimation via Bayesian subgroup analysis. The FDA has also held a symposium and a workshop on heterogeneous treatment effects, each including discussions and presentations on shrinkage estimation using Bayesian hierarchical models [2,3]. Office of Biostatistics statisticians published a manuscript on “Bayesian Hierarchical Models for Subgroup Analysis,” providing various examples and advice for constructing such models. The Center for Drug Evaluation and Research, on April 15, 2024, rolled out three demonstration programs including one on Bayesian Supplemental Analysis [4]. One Bayesian statistical analysis plan posted on the FDA website (https://www.fda.gov/media/178887/download?attachment) is on using a Bayesian hierarchical model to simultaneously determine estimated treatment effects (via hazard ratios) across four regions for a time-to-event endpoint [5]. Data from all regions are used in estimating each region-specific hazard ratio.
Talk 2: David Ohlssen (Novartis) “Bayesian shrinkage estimation for subgroup analysis in clinical trials: Examining the critical aspects”
The authors discussed traditional Bayesian shrinkage methods (based on a random-effects structure) and contrasted it with full treatment effect by subgroup stratification assuming a single overall treatment effect. While simple shrinkage provides a logical compromise between stratification and a single pooled effect, it still leads to a couple of issues that were highlighted in the talk: First, the importance of the selection of prior on the variance among the subgroup treatment effects (particularly with small number of groups), and second, the desire to look at multiple factors (e.g. sex, region and disease severity) which each lead to a separate set of disjoint subgroups (often a small number). The subgroups associated with different factors overlap in terms of patients, and this is not accounted for when applying simple shrinkage multiple times. As an alternative, the authors suggested building a regularized global model for prediction, including many treatment by covariate interactions, and using this model as a basis for multiple subgroup analyses as in Wolbers et al (2024) [6]. To achieve regularization within the Bayesian framework special priors such as the horseshoe or R2D2 (applied by the authors) priors must be used. The presentation concluded with a comparison of approaches based on two phase III asthma studies with identical protocols. A neutral comparison was performed by leaving one study out to assess performance. In general, the global model approach worked well and was suggested as a valuable approach to test in the FDAs Bayesian supplementary analysis demonstration project.
Discussion: Chenguang Wang (Regeneron)
The session, featuring two excellent presentations, is highly relevant for those considering subgroup analyses, particularly through a Bayesian approach. Dr. Rothmann’s presentation focused on applying Bayesian subgroup analyses in Drug Trials Snapshots (DTS), aiming to make demographic data more available and transparent. Typically, subgroups in DTS are defined by sex, race, age, and ethnicity, and the number of subgroups is usually small. Dr. Rothmann explained how Bayesian priors affect the results and recommended priors that produce robust outcomes from a regulatory perspective. In contrast, Dr. Ohlssen’s presentation leaned towards exploratory scenarios, which often consider a large number of subgroups. This necessitates the introduction of certain model structures of the subgroup factors, such as the global outcome model from Wolbers et al. (2024) [6], and regularization-type priors like the R2D2 prior. Dr. Ohlssen emphasized the importance of a well-developed workflow for conducting Bayesian subgroup analysis in exploratory settings.
It is encouraging to see that Bayesian subgroup analysis, an “innovative statistical approach,” is making an impact from both regulatory and industry perspectives. The different strategies presented by Drs. Rothmann and Ohlssen highlight the flexibility of this Bayesian method and exemplify the significant contributions statisticians can make to drug development.
References
FDA Impact Story “Using innovative statistical approaches to provide the most reliable treatment outcomes information to patients and clinicians: Using Bayesian Hierarchical Models to Improve Our Understanding of Drug Effects” Available at impact-story-using-innovative-statistical-approaches-provide-most-reliable-treatment-outcomes. Accessed October 8, 2024.
Nov 28, 2018, Symposium of Assessing and Communicating Heterogeneity of Treatment Effects for Patient Subpopulations: Challenges and Opportunities. Agenda, Slides and Recording at https://www.jhsph.edu/research/centers-and-institutes/center-of-excellence-in-regulatory-science-and-innovation/news-and-events/Critical-Issues-in-Heterogeneity-of-Treatment-Effect.html. Accessed October 8, 2024.
Nov 30 - Dec 1, 2020, Workshop on Heterogeneity of Treatment Effects in Clinical Trials: Methods and Innovations. Agenda and Recording at https://mrctcenter.org/news-events/heterogeneity-of-treatment-effects-in-clinical-trials-methods-and-innovations/ - 1602863324215-1289c9d5-a82a. Accessed October 8, 2024.
Wang, Y., Tu, W., Koh, W., Travis, J., Abugov, R., Hamilton, K., Zheng, M., Crackel, R., Bonangelino, P., Rothmann, M. Bayesian Hierarchical Models for Subgroup Analysis. Pharmaceutical Statistics. 2024; 0 1-19. https://doi.org/10.1002/pst.2424. Accessed October 8, 2024.
Example Statistical Analysis Plan for Bayesian Subgroup Analysis: Sharing of Information Across Subgroups. Available at https://www.fda.gov/media/178887/download?attachment. Accessed October 8, 2024.
Wolbers, M., Rabuñal, M. V., Li, K., Rufibach, K., & Bové, D. S. (2024). Using shrinkage methods to estimate treatment effects in overlapping subgroups in randomized clinical trials with a time-to-event endpoint. arXiv preprint arXiv:2407.11729. https://arxiv.org/abs/2407.11729
Appendix
Session Abstract
Due to automatic borrowing of strength across groups, Bayesian shrinkage methods result in more precise subgroup treatment effect estimates. Mathematically this typically results in a favorable bias variance tradeoff when compared to a traditional stratified subgroup analysis. More importantly from a practical perspective the shrunken estimates should provide reasonable estimates of the treatment effect a future patient would expect, along with the uncertainty surrounding these effects. Such an approach helps to avoid over-interpretation of "random high" trends that are highlighted by forest plots displaying fully stratified subgroup analysis and can be harmful for public health and individual patients (Sleight, 2000). The idea of using Bayesian methods for subgroup analyses dates back at least 40 years. A review of the initial work is provided by Jones et al (2011). Many of the methods discussed in this paper were implemented in the beanz R package (Wang et al (2018)). In 2019 the FDA (FDA, 2019) published an impact story showing how it uses Bayesian shrinkage estimation in publishing drug trials snapshots. Given the increased understanding and regulatory utilization of these methods for specific cases, what is holding us back from routine application? It is the case that some practical considerations remain unresolved. For example, which hierarchical variance prior to use and on what scale to impose shrinkage of estimates. Further, a key limitation of earlier work, is that Bayesian subgroup shrinkage analyses only allow shrinkage across disjoint, non-overlapping subgroups. In most clinical trials, however, several variables are considered of interest for subgroup analyses. Running separate shrinkage analyses for each subgroup variable could be one option, but then the correlation among the separate analyses is not accounted for and the set of separate models provide limited insight on which variables are the drivers of treatment effect heterogeneity. In this conference session, our aim is to present practical insights gained from applying existing Bayesian shrinkage methods and examine the latest advancements in this field aimed at overcoming some of the key challenges.
References
FDA (2019) "Impact Story: Using innovative statistical approaches to provide the most reliable treatment outcomes information to patients and clinicians" https://www.fda.gov/drugs/regulatory-science-action/impact-story-using-innovative-statistical-approaches-provide-most-reliable-treatment-outcomes
Jones, H. E., Ohlssen, D. I., Neuenschwander, B., Racine, A., and Branson,M. (2011), "Bayesian Models for Subgroup Analysis in Clinical Trials," Clinical Trials, 8, 129-143
Sleight, P. (2000) Debate: Subgroup analyses in clinical trials: fun to look at - but don't believe them!. Trials 1, 25
Wang, C., Louis, T. A., Henderson, N. C., Weiss, C. O., and Varadhan, R. (2018). "beanz: an R package for Bayesian analysis of heterogeneous treatment effects with a graphical user interface". Journal of Statistical Software, 85, 1-31.