2021 FDA Science Forum
Physiological Model Selection for in silico Evaluation of Closed-Loop Medical Devices
- Authors:
- Center:
-
Contributing OfficeCenter for Devices and Radiological Health
Abstract
Introduction
Physiological closed-loop controlled (PCLC) medical devices are complex systems integrating one or more medical devices with a patient's physiology through closed-loop control algorithms; introducing many failure modes and parameters that impact performance. Credible mathematical models have been constructed and used throughout the development and evaluation phases of a PCLC device. Uncertainties about the fidelity of these models need to be addressed before achieving a reliable PCLC evaluation. To identify the best candidate model toward in silico evaluation of PCLC devices, this research develops tools for properly assessing model accuracy and establishes fundamental measures for evaluating predictive capability performance across multiple models.
Methods
As a case study, a refined model of blood volume (BV) response was built by expanding an original model we developed in our prior work. We used the experimental data collected from 16 sheep undergoing hemorrhage and fluid resuscitation. First, the calibration performance of the two candidate models, i.e., original and refined, was compared using root-mean-square error (RMSE), Akiake information criterion (AIC), and a new multi-dimensional approach utilizing normalized features extracted from the fitting error. Second, predictive capability of the two models was compared under three different scenarios: prediction of subject-specific steady state BV response, subject-specific transient BV response to hemorrhage perturbation, and leave-one-out inter-subject BV response.
Results
The refined model demonstrated a significant calibration performance improvement in terms of RMSE (9%, P = 0.03) and multi-dimensional measure (48%, P = 0.02), while a comparable AIC between the two models verified that the enhanced calibration performance in the refined model is not due to data over-fitting. Furthermore, results indicated enhanced accuracy and predictive capability performance for the refined model with significantly larger proportion of measurements that were within the prediction envelop in the transient and leave-one-out prediction scenarios (P < 0.02).
Conclusion
This study helps to identify and merge new methods for credibility assessment and physiological model selection, leading to a more efficient process for PCLC medical device evaluation. We showed that between the two candidate BV models, the refined model performed better or at least comparable in all three scenarios. Thus, the refined model could offer more credibility toward PCLC medical device assessment. The developed pathway can be extended for model selection in other physiological domains.
