1	Details of FDA’s Pilot Risk-Ranking Model for GMP Inspections Nga Tran, Dr.P.H., Exponent Brian J. Hasselbalch, FDA July 2004
2	Application of Risk Ranking Regulatory programs: EPA, Cal EPA USDA Risk management DoD Industry
3
4	EPA Waste Minimization Prioritization Tool (WMPT) A regulatory decision tool Foundation for the EPA’s Resource Conservation and Recovery Act Persistent, Bioaccumulative, and Toxic (PBT) List of chemicals. Framework of expert judgment Identify chemicals or emissions of potential concern Using key physical-chemical properties and associated cutoff criteria.
5
6
7
8	DoD Comparative risk pre-deployment Rank locations based on potential chemical risks Risk score is a function of volume and inherent chemical toxicity Comparative risk during deployment Ranking and comparing disparate risks Quick/simple tools
9
10
11
12
13	Consumer Product Example Ranking to target products for which additional toxicity testing of High Production Volume chemical is needed. Laundry detergent, consumer care products, etc.. Many products, limited resources Ranking based on exposure to consumer products “High end” product use frequency (e.g., number of times use per day) as surrogate for exposure
14
15
16	Food Microbiological Hazard Example Van Gerwen et al (2000) SIEFE: Stepwise and Interactive Evaluation of Food Safety by an Expert System. A decision support system for microbiological risk assessment for food products and their production processes Risks are first assessed broadly, using order of magnitude estimates, i.e. level 1 risk assessment. Characteristic numbers are used to quantitatively characterize microbial behavior during the production process to highlight the major risk-determining phenomena
17
18	Food risk ranking model (Ross and Sumner, 2002) Tool to compare food-borne risks for ranking and prioritizing risks from diverse sources. 11 questions relating to 3 main factors: severity of hazard likelihood of disease causing dose of hazard being present in a meal probability of exposure to the hazard
19
20
21
22	USDA- Food Safety and Inspection Service "Inspector Optimization System" (IOS) Basis of assigning the processing inspector workforce to provide for greater flexibility in targeting inspection resources to areas of greater public health risk and concern Report to Congress on Risk-Based Inspection to the United States House and Senate Appropriations Committees, March 2, 2001 Each plant receives a Food Safety Hazard Coefficient (HC) Inherent hazards Expert Elicitation to rank inherent hazards
23
24
25
26
27	The CDER-OC process Began one year ago Internal experts from CDER, CVM, CBER, and ORA (Risk Mgmt & Workplan WG) Generated a list of risk factors that are descriptors of site risks for consideration in development of site selection model Qualitatively valued: High, Medium, Low
28	Examples of risk factors
29	Develop Risk Ranking Framework High level risk management begins with organization Categorized list of risk factors into 3 components Product Process Facility
30	Categorization of risk factors, e.g.
31
32	Drilling Down to Component Factors Selected factors for product, process and facility components Assigned weights Empirically derived Expert judgment Developed logical algorithm to combine factors to calculate a site score – The Model
33	"Drilling Down and Populating the..." Drilling Down and Populating the Product, Facility, and Process Components
34
35	Product Component Data Data source: FACTS (Field Accomplishments and Compliance Tracking System) Site information/identifier Product codes CDER Recall Database Product codes Rx/OTC and Classification of Hazard Years: Recall: 1997-2004 Product: 2000-2004
36	Recall Weight Matrix
37	Product Categorization Need to link recall data to sites Indirect linkage -- correlate product codes in recall data with product codes in site database Recall weights assigned to Recall-FACTs harmonized product codes
38	Examples of Product Code Correlation
39
40	Facility Component Data Data source: FACTS (Field Accomplishments and Compliance Tracking System) Years: 2000-2004 All sites, foreign and domestic
41
42	Process Component Data No data, but do have staff with experience and knowledge \ expert elicitation survey Elicitation survey drafting began Nov. 2003 Inter-Center/ORA workgroup effort ORA; CVM; CBER; CDER; CDER; OC
43	Expert Elicitation Survey Challenges What are the relevant process-related risk factors? (i.e., sources of variability and poor quality) What, if any, unit operations are more liable to a loss of control or at risk to contamination? What products? Answers are product dependent Large number of products Identify “mutually exclusive” category of products -- tradeoffs in number of product categories and responder burden
44
45	Defining product types and unit operations Product types Several CDER/agency coding schemes; too much Types selected are representative of most product types in the market Codes grouped into product types with similar unit ops Distinguish high/low actives Unit operations WG member knowledge + Remington: Science and Practice of Pharmacy, 20^th edition Modern Pharmaceutics, 3^rd edition Pharmaceutical Process Validation, 3^rd edition ISPE Baseline Pharmaceutical Engineering Guide, vol. 2, Oral Solid Dosage Forms (1^st edition, 1998)
46
47
48	PROCESS RISK FACTORS A) Process control To what degree does this unit of operation contribute to variability in quality of the final product? 2) How difficult is it to maintain this unit of operation in a state of control? 3) If a problem does occur, how reliable are the current detection methods? B) Contamination 4) Is this unit of operation more or less vulnerable to contamination from previous product? 5) Is this unit of operation more or less vulnerable to contamination from the environment?
49
50
51	Survey Status Delivery By e-mail 50 FDA experts participated reviewers from CDER senior CDER compliance staff senior ORA field staff 90% response rate Results being analyzed
52	Data analysis Exploratory Data Analysis (EDA) Product rank consistent with unit of operation drill down data – correlation Developing process weights based on unit of operation drill down survey data K-mean cluster analysis Average rank per product category per question a) Process controls: questions 1, 2 and 3 b) Contamination potential: questions 4 and 5 Coefficient of Variance (CV) Weighted average rank per product category per question Principal component and fuzzy arithmetic for expert categorical data analysis
53
54	Plain language summary A site will tend to be less frequently selected for inspection if it: has been inspected recently and/or relatively few previous violations of GMPs and/or a smaller volume of product (facility weight); makes non-sterile, OTC drugs, and/or other product types that are not associated with a high frequency of serious recalls (product weight); makes products estimated to be relatively straightforward to manufacture and not vulnerable to contamination (process weight)
55	Current and Future Risk Factors in Site Score
56	Questions to the Subcommittee Can you identify alternative approaches that would systematically prioritize manufacturing sites for GMP inspections? In what areas would additional data provide the most value added in prioritizing manufacturing sites for GMP inspections? Are there other metrics that should be incorporated, e.g., measuring process control?