U.S. flag An official website of the United States government

On Oct. 1, 2024, the FDA began implementing a reorganization impacting many parts of the agency. We are in the process of updating FDA.gov content to reflect these changes.

U.S. Food and Drug Administration
  1. Home
  2. Science & Research
  3. Science and Research Special Topics
  4. Advancing Regulatory Science
  5. The Digital Variome: Understanding the Implications of Digital Tools on Health
  1. Advancing Regulatory Science

The Digital Variome: Understanding the Implications of Digital Tools on Health

CERSI Collaborators: University of California at San Francisco (UCSF): Andrew Auerbach MD MPH (UCSF, Variome and DOVeS); Benjamin Rosner MD PhD (UCSF, Variome and DOVeS co-PI); Stanford Univerisity (Stanford): Matthew Horridge PhD (Stanford, DOVeS)

FDA Collaborators: Center for Devices and Radiological Health(CDRH): Bakul Patel, MS, MBA (Formerly of CDRH); Vinay Pai, PhD; Catherine Bahr; Leeda Rashid, MD, MPH, ABFM; Arti Tandon, PhD; Charlie Yongpravat, PhD; Anindita Saha, PhD

CERSI In-Kind Collaborators: Stanford Univerisity (Stanford): Mark Musen, PhD (Stanford, DOVeS)

Project Start Date: October 12, 2021

Regulatory Science Challenge

There is a consistent need to research and develop the methods used to ensure the quality and safety of FDA-regulated products. Research and development in this area helps FDA employ scientifically valid approaches for combining patient input and data from multiple sources. These 'real world' data insights are key to informing regulatory decision-making both for traditionally regulated products (e.g. drugs and devices) as well as for new and emerging products such as digital health tools. Furthermore, as the FDA considers new products for regulatory approval, they may examine whether these new products are “substantially equivalent” to previously approved products. The wave of new digital health products creates regulatory decision challenges that need to be informed by real world data and data that help identify the degree to which products may be similar.

Project Description and Goals

The Digital Variome project extends work that is ongoing as part of our overarching CERSI project Developing Frameworks and Tools for Integration of Digital Health Tools into Clinical Practice, a national network of leading academic medical centers, researchers, and innovators working to identify how real world measures and data can be used across types of software used in health, and the eventual data sources required to carry out real world performance measurement and post-market surveillance of digital health tools (DHTs). ADviCE identified several challenges to DHT adoption: (1) Variable definitions of which DHTs are relevant to clinical care delivery; (2) Lack of consistent, common terms to describe DHTs during selection, (3) Wide variability in how health systems integrate DHTs into practice and, (4) Lack of a framework and tools to evaluate DHTs’ real-world performance through post-market surveillance.

The ADviCE project in turn framed the goals of the Variome proposal, which focused on identifying data sources and potential partnerships needed to create a learning health collaboration that might leverage tools such as NEST or resources (e.g., PCORnet, or payor data) to provide data needed to carry out post-market surveillance of DHTs. Few of these data networks or partnerships could gather information needed for DHT post-market surveillance, so investigators turned their attention to tools which would both facilitate efficient specification of DHT characteristics while also being flexibly able to accommodate measures that might vary between DHTs even though applicable to similar patients or health systems.

With this realization, the research team extended their Priority Measurement framework and expanded it to represent a range of potential metrics applicable to real world performance. Investigators built on their consensus work from Developing Frameworks and Tools for Integration of Digital Health Tools into Clinical Practice to identify specific domains and measures relevant to each broad domain. For example, within the area of Product Performance  Cybersecurity, investigators developed subdomains where metric identification was recognized as a key next step. Not surprisingly, a wide range of potential measures were identified. For example, each of the Measure Concepts for Real World Health might have dozens or even hundreds of patient- or population-specific metrics that are supported by evidence, are broadly used, or both.

Domain Subdomain Priority Measure Concept
Real World Health Health Benefits*

Gathers data in a valid and unbiased way
Gathers positive and negative health outcomes
Gathers positive and negative aspects of patient reported health or patient generated data
Gathers information relevant to defining healthcare value
Clinical Safety

Gathers safety and harm events
14971 categories, risk management standards for medical devices.
Systems in place to report and mitigate harms when detected
Real World Usage

Gathers information on ‘off label’ use
User (or system or sub-group) retention rates over time
User costs to adopt and maintain

This realization led to development of the Digital medicine Outcomes Value Set (DOVeS), as a powerful and flexible approach to classifying digital health tools according to key features and important clinical outcomes identified by our work to this point.

DOVeS was blueprinted using Protégé software using input from research collaborators and professional ontologists so that it permits flexible expansion as outcomes or population definitions change and technology advances. DOVeS was then tested and validated against real DHT and company characteristics to yield a working prototype that facilitates search and display of data using the overall ADviCE/Variome approach. DOVeS has been published on BioPortal and is publicly available for broad use.

DOVeS has the potential to be scaled up to include a broader and more representative sample of real-world digital health tools, accommodate new technologies (e.g. large language lodels (LLMs)), while also being tested for usability and feasibility as a practical framework for use by health systems, vendors, and regulators use.

Research Outcomes/Results

There are several outcomes to date associated with the development of the Variome project and DOVeS Ontology. The first is that the DOVeS ontology has been expanded substantially over the course of this support, informed by real world digital health outcomes gleaned from industry and academic experts. The second is that DOVeS has been made publicly available on the BioPortal website so that a community of digital health experts may continue to contribute to it over time. The third is that a prototype user interface overlying DOVeS has been created (only a non-functional wireframe was originally proposed) leading to functional demonstrations that show the power and value of DOVeS in identifying tools based on common outcomes. Fourth, several public presentations of DOVeS have been made. Finally, a peer reviewed publication on the development of DOVeS is forthcoming and will help disseminate awareness of the ontology and its value. In the future, investigators hope to convert the prototype front end user interface into a robust platform capable of supporting regulatory insights as well as health system leader inquiries and decisions about digital health tools.

Research Impacts

This project enhances foundational requirements for regulatory science research by providing the FDA and other stakeholders with a new way to categorize and identify digital health tools based on outcomes they influence. This is particularly valuable to enable more appropriate apples-to-apples comparison of digital health tools that influence similar outcomes which could be valuable for "substantial equivalence" assessment as well as both superiority and non-inferiority considerations. The ontology is also particularly valuable for ongoing post-market surveillance.

 

 

Back to Top