• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

Food

  • Print
  • Share
  • E-mail

Assessment of the Third-Party Certification Pilot for Aquacultured Shrimp


July 2011


U.S. Food and Drug Administration

Center for Food Safety and Applied Nutrition
Office of Regulatory Affairs


Table of Contents


Introduction

The challenges of ensuring the safety and security of the U.S. food supply are greater now than ever before.  The U.S. food supply is truly globalized.  Foods are imported into the U.S. from more than 200 countries, dependencies, and areas of special sovereignty and include both raw ingredients and finished, ready to eat products.  More than 240,000 foreign food facilities are registered with FDA.  New food sources and increasingly complex food production and distribution methods present challenges to FDA's oversight of this global supply chain.  Additionally, new and evolving food borne disease agents leads to the potential for increased foodborne illness.  These challenges are occurring at a time of ongoing concerns about resources to oversee the food supply in the United States.  For these and other reasons, various approaches have been considered to address the safety of imported food.

Third-party certification has been discussed both internally at FDA and in Congress for the last several years.  In a 2004 seafood safety report, the then General Accounting Office recommended that FDA explore the possibility of certifying third-parties to conduct inspections of foreign seafood processors and domestic importers, similar to FDA's third-party inspection program for devices.[1] 

In June 2010, a committee of experts convened by the Institute of Medicine and the National Research Council (IOM/NRC committee) released a report examining gaps in public health protection afforded by the farm-to-table food safety system under FDA's purview and identifying opportunities to fill those gaps.[2]  The IOM/NRC committee concluded that FDA needs to address barriers to improving the efficiency of inspections by, among other things, exploring third-party auditing of food facilities as an alternative model for measuring compliance.  The IOM/NRC committee's report, entitled Enhancing Food Safety: The Role of the Food and Drug Administration, specifically recommended as follows:

Recommendation 8-3: The FDA should fully consider the implications of accepting inspection data from an auditing program in which third-party auditors would inspect facilities for compliance with food safety regulatory requirements.  If this approach is utilized, the FDA should set minimum standards for such auditors and audits, with oversight and implementation being assigned to an accreditation and standards body.

FDA is evaluating this IOM/NRC recommendation in conjunction with the information and data gleaned from a two-year, third-party certification pilot project the agency recently completed.  This report summarizes findings and conclusions from FDA's Third-Party Certification Pilot for Aquacultured Shrimp (Shrimp Pilot or Pilot).

The Shrimp Pilot was designed and conducted, and its results analyzed before enactment of the Food Safety Modernization Act (FSMA) in 2011.  As such, the context of the pilot project was based upon a voluntary approach for industry to participate in a third party system with no explicit legislative authority or mandate for FDA to operate such a program.  With the passage of FSMA, the new legislation requires FDA to establish a third-party certification program for foods that involves recognition of accreditation bodies which, in turn, accredit third-party auditors and possible direct accreditation by FDA of third-party auditors.  See 21 U.S.C. § 384d.  In this Shrimp Pilot, the role FDA played was analogous to the role traditionally played by an accreditation body.  While FDA anticipates that this Shrimp Pilot will provide valuable insight as the agency moves forward in the development of a third party certification program for foods, the observations and recommendations contained in this report have not been specifically adjusted to take into account the new third-party accreditation mandate in FSMA.

Background

On April 2, 2008, FDA published a Federal Register (FR) notice requesting comments on the benefits, obstacles, and availability of third-party certification programs for foods and animal feeds.[3]  To explore the potential benefits and challenges of using third-party certification, on July 10, 2008, FDA published two FR notices[4]; one announcing the availability of FDA's draft Guidance for Industry on Voluntary Third-Party Certification Programs for Foods and Feeds, and the other inviting third-party certification bodies (CBs) to participate in a pilot of voluntary third-party certification of aquacultured shrimp.  The goal of the Pilot was to gain knowledge and experience with third-party certification to assist FDA in evaluating the utility and feasibility of using third-party certification programs as a means of leveraging additional resources for foreign inspection, sampling, and data collection. This document provides:  an overview of FDA's approach, including a discussion of FDA's Final Guidance for Third-Party Certification for Foods and Feeds[5] (published on January 15, 2009), observations on the use of third-party certification in the shrimp industry, and a discussion of the potential application of Pilot findings to broader use by FDA of third-party certification programs.

What is Third-Party Certification?

Third-party certification is the procedure by which a CB – an organization other than the establishment seeking certification and other than FDA (or other governmental entity acting under FDA authority) – provides assurance that an establishment conforms to specified certification criteria, which may vary by product and client.  To determine conformance, the CB conducts a range of audit activities (e.g., onsite examinations of establishments and laboratory testing of product samples).  When the CB auditors determine that an establishment meets the specified criteria, the CB will grant certification for the particular product types produced, manufactured, processed, or held by the establishment.  A third-party can be a private entity or a Federal, State, local or foreign government authority that is not conducting inspections under FDA authority and should be independent of (i.e., free from conflict of interest or financial ties to) the establishments it audits for certification.

How Did FDA Select an Industry for the Pilot?

In determining which industry to use for the Pilot program, FDA sought to identify a product/industry that met the following criteria:

  1. 1)  A product with a well established safety standard against which performance could be measured;
  2. 2)  An industry that has significant experience with third-party certification programs;
  3. 3)  A product with a history of compliance issues (e.g., contamination) to enable evaluation of the ability and degree to which certification programs are able to detect conditions that could lead to such contamination; and
  4. 4)  A product with a high volume of imports and a broad range of sources

Why Use Aquacultured Shrimp for the Pilot?

FDA chose to focus the Pilot on aquacultured shrimp for several reasons:

Certification criteria are those criteria used by the CB during an audit to determine whether a establishment should receive certification. Shrimp Pilot Certification Criteria should, at a minimum, include applicable FDA requirements.
  • Existence of a regulatory food safety standard (Seafood Hazard Analysis and Critical Control Point or HACCP) for the shrimp industry:  FDA wanted a product with a well-established safety standard that could be used by a CB to determine whether an establishment should receive certification.  The Seafood HACCP regulation[6] was viewed as an ideal candidate for use as certification criteria because it is well defined by FDA and has been in effect and enforced since 1997.   
  • Established industry experience with third-party certification: FDA was interested in working with an industry that has had significant experience with third-party certification programs.
  • History of contamination problems: FDA has observed microbial contamination and unapproved drug uses in aquacultured shrimp.  FDA believed that focusing on a product with a history of these types of contamination problems would enable an assessment of the ability of third-party auditors to detect conditions that could lead to these types of contamination.
  • High volume of imports and a broad range of sources: Aquacultured shrimp is imported in large and ever-increasing volumes to the U.S. (see Figure 1 below).  Additionally, aquacultured shrimp is produced in a number of countries all over the world.  FDA hoped that the large volume and variety of sources would provide an opportunity to assess the challenges associated with third-party certification of diverse foreign processing operations.

 A line graph showing the increase, in thousands of pounds, of shrimp imported into the United Sates from 1991 to 2006.  The graph shows the volume increased from 540,252 in 1991 to 1,306,623 in 2006.

Figure 1: U.S. aquacultured shrimp imports, in volume of 1,000 lbs.

USDA Economic Research Survey, October 2009.

What is Aquaculture?

For purposes of the Pilot, FDA used the definition of aquaculture provided in the Food and Agricultural Organization (FAO) CWP Handbook of Fishery Statistical Standards:

"Aquaculture is the farming of aquatic organisms: fish, mollusks, crustaceans, aquatic plants, crocodiles, alligators, turtles, and amphibians.  Farming implies some form of intervention in the rearing process to enhance production, such as regular stocking, feeding, protection from predators, etc. Farming also implies individual or corporate ownership of the stock being cultivated.  For statistical purposes, aquatic organisms which are harvested by an individual or corporate body which has owned them throughout their rearing period contribute to aquaculture, while aquatic organisms which are exploitable by the public as a common property resource, with or without appropriate licenses, are the harvest of capture fisheries."[7]

Pilot Program Structure

The announcement for the Shrimp Pilot indicated the Pilot would be conducted in two phases, with the first phase encompassing the solicitation of Pilot participants and paper audits to determine whether applicants had the attributes described in the Pilot announcement, and the second phase, which would include selection of CBs to participate in the Pilot and the onsite audits by FDA of the CBs by accompanying certain CB auditors during certification and /or recertification inspections.

In practice, the Pilot actually took place in the following three phases:

  • Phase 1:  Develop and Initiate the Pilot:   This phase was conducted to lay the policy groundwork for the Pilot and initiate the application and selection processes for participation in the Pilot.  It included the preparation and publication of three FR notices and the draft and final guidance on FDA Third-Party Certification Programs for Foods and Feeds.
  •  Phase 2:  Assess Certification Body Program Attributes:  This phase focused on gathering additional information about each prospective Pilot CB's certification program to assess the degree of conformance with FDA defined attributes needed for certification programs.[8]  FDA reviewed self-assessments conducted by each CB, and FDA also conducted an onsite review of the certification program at the headquarters of each CB accepted into the Pilot program.
  • Phase 3:  Audit Performance of Certification Bodies:  In this phase, FDA observed CB auditors conducting audits at shrimp processors and farms.  FDA also conducted its own audits of laboratories that the CBs used to support their program functions.

Phase 1:  Develop and Initiate the Pilot

Phase 1 of the Shrimp Pilot included the development and publication of the draft and final Guidance for Industry- Voluntary Third party Certification Programs for Foods and Feeds and the announcement of the Pilot program requesting interested CBs to apply.  Phase 1 culminated in the formal selection and announcement of the Pilot participants.  The following timeline illustrates the key Phase 1 milestones of the Shrimp Pilot, which took place from April 2008 through January 2009.

 A timeline showing the milestones in Phase 1 of the Shrimp Pilot beginning with the publication of the Federal Register Notice for Third-Party Certification at FDA on April 2, 2008; publication of the draft Guidance on Third- Party Certification and announcement of the Pilot on July 10, 2008; the application deadline of August 31, 2008; announcement of the Pilot participants on December 2, 2008, and publication of the Final Third-Party Guidance on January 15, 2009.

Figure 2: Phase 1 Milestones

A. Pilot Development

Prior to announcing the Shrimp Pilot, in April 2008, FDA sought public comment about the current practices of third-party certification programs for foods and feeds and input on the design and development of third-party certification programs by FDA.  The comments received provided FDA with insight into industry and public expectations and concerns with third-party certification and informed the development of the draft guidance during Phase 1 of the Pilot.

On July 10, 2008, FDA issued Draft Guidance for Industry on Voluntary Third-Party Certification Programs for Food and Feeds.  In the same issue of the FR FDA also announced the initiation of the Shrimp Pilot, requesting interested CBs to submit applications for participation in the Pilot.  

B. Application Process

By the August 31, 2008 deadline, 45 days after the Pilot was announced, six CBs had submitted applications to FDA.  One CB provided only a letter of interest and was subsequently removed from consideration.  After review of the applications, FDA publicly announced the six CBs as the official Pilot participants on December 2, 2008.

The CBs represented a variety of sectors, including non-profit, private for-profit, and public sectors.  A variety of countries were also represented, with two of the six CBs based in the United States.

Table 1 – Certification Bodies Participating in Pilot
Certification Body Headquarters
Office Locations
Global Aquaculture Alliance/Aquaculture Certification Council Inc. United States
Bureau Veritas Group Company France
Inspectorate America Corporation Argentina
SGS United States Testing Company United Kingdom
Thailand Department of Fisheries Thailand
Seafood Inspection Service, National Marine
Fisheries Services, National Oceanic and
Atmospheric Administration, U.S. Department of Commerce
United States

Observations:

  • While CBs that applied to the Pilot appeared to at least have elements of the majority of requirements identified in the Pilot announcement, the amount of supporting documentation submitted with the applications varied.  As a result, extensive follow-up with all of the CBs was needed in order to gather sufficient information to determine whether the applicant should be accepted into the Pilot.
  • After accepting CBs for the Shrimp Pilot, FDA discovered that some of the CB certification programs were using something other than FDA's seafood HACCP regulations as certification criteria, e.g. British Retail Consortium (BRC) standards.[9]
  • The description of the Pilot in the FR announcement stated that FDA intended to conduct onsite audits of the third-party certification programs by accompanying certain auditors during certification and recertification audits. Several CBs assumed that FDA was interested in only visiting their U.S. base of operations. FDA had to clarify with Pilot applicants its intention to also visit the CB's international headquarters.

Future Considerations:

Based on FDA's experience with the application process, it is apparent that in any future program, FDA should be clearer in its expectations for the amount and type of information needed to adequately evaluate a firm's application.

C. Final Guidance for Third-Party Certification

FDA published the Final Guidance for Industry – Voluntary Third-Party Certification Programs for Foods and Feeds (the Guidance) on January 15, 2009 (http://www.fda.gov/RegulatoryInformation/Guidances/ucm125431.htm). FDA received a total of 27 public comments in response to the draft guidance.  In general, the comments supported third-party certification as long as it was a voluntary program.  However, some comments indicated that consumers were skeptical that third-party certification could provide the same assurances of safety as would FDA inspection.  The final Guidance addressed many of the comments FDA received.  The most significant change was a shift in focus from product specific certification to program attributes.  The table below is a summary of the major changes between the draft and final Guidance.

Table 2 – Key Changes Between Draft and Final Third Party Certification Guidance
Area of Guidance Draft Guidance Final Guidance
Standards vs. Attributes Included certification criteria that a food/feed firm should meet. Does not include certification criteria; focuses on attributes that the certification body should meet.
Auditor Definition A person who is responsible for oversight of an inspector, inspection or certification program. For the purposes of this document, an auditor will refer to a person acting for the Certification body during a self-assessment or FDA or an accreditation body during its assessment of the certification program. A person acting for the Certification body who conducts audits and makes a determination of the degree to which certification criteria have been met by a firm for a particular product type
Certification Program Assessment Not mentioned A systematic approach by FDA to assess a CB's conformance with the attributes laid out in the final guidance
Notification to FDA Safety issues, fraud, and criminal acts Safety issues, withdrawing certification, changes to certification program
Recertification Based on risk Every 2 years; annually for high-risk products/processes


Phase 2: Assess Certification Body Program Attributes

Phase 2 of the Shrimp Pilot focused on gathering information about each CB's certification program to assess the degree of conformance with FDA defined attributes needed for certification programs.  FDA reviewed, via paper audit, self-assessments conducted by each CB.  FDA also conducted an onsite review of the certification program at the headquarters of each CB.  The following timeline illustrates the key Phase 2 milestones of the Shrimp Pilot which took place from February 2009 through July 2009.

 A timeline showing milestones in Phase 2 of the Shrimp Pilot beginning with providing the self-assessment checklist to the Certifying Bodies on February 6, 2009; the deadline for returning the self-assessment checklist of March 15, 2009; the start of scheduling for on-site assessments of the Certifying Bodies on March 17, 2009; the beginning of the actual on-site assessments on May 4, 2009; and the completion of all the on-site assessments of the Certifying Bodies on July 3, 2009.

Figure 3. Phase 2 Milestones

FDA used the eleven attributes from the final Guidance to assess the CBs' degree of conformance with FDA defined attributes needed for certification programs (see Appendix A).   Below is a description of the eleven attributes.

Attributes for Third-Party Certification Programs

  1. 1)  Authority of the Certification Body – A certification body should enter into a contract or other arrangement with an establishment that grants the certifying body authority to ensure that the establishments and processes they are hired to certify meet the certification criteria.  The certification criteria should at least include the U.S. regulatory requirements that apply to the food being certified.
  2. 2)  Qualification and Training for Auditors – All auditors should understand the food safety issues related to the processes and products that they audit.  This should include knowledge and understanding of current certification criteria (including FDA regulations) and a process to help ensure that this knowledge and understanding are kept up to date.
  3. 3)  Elements of an Effective Audit Program – The certification body should have procedures that ensure that its auditors consistently meet established standards for a high quality audit, as well as consistently audit for conformance with certification criteria when auditing an establishment.
    Attributes are characteristics of a CB and its certification program that provide a reliable reflection that the certification program will ensure that foods from an establishment certified by the CB meet applicable FDA requirements, as well as other certification criteria.
  4. 4)  Quality Assurance Program for Audits and Auditors – The certification body should implement a quality assurance program that monitors its auditors (including subcontractors), audits, and sample collection processes for consistency and competency, to identify areas that need improvement, and to quickly execute appropriate corrective actions when problems are found.
  5. 5)  Compliance and Corrective Action – The certification body should have strategies, procedures, and actions to ensure that the establishments and processes for particular product types it certifies comply with FDA laws and regulations and otherwise meet certification criteria and take appropriate steps when there is non-compliance.
  6. 6)  Industry Relations – The certification body, at a minimum, should provide establishments seeking certification with information about current FDA laws, regulations and guidances, and the certification criteria.  The certification body may also conduct activities that foster communication and information exchange among regulators, industry, academia, and consumer representatives on product safety and security.
  7. 7)  Resources – The third-party certification body should have sufficient resources, such as equipment and infrastructure, etc. to accomplish the elements of the certification program (as described in FDA guidance).
  8. 8)  Self Assessmentof the Overall Certification Program – The certification body should conduct an initial self-assessment to assess its performance in relation to the attributes (in FDA guidance), including identification of the strengths and weaknesses of its certification program.  Subsequent self-assessments should be done at least every 36 months thereafter and after any significant changes to the certification program.
  9. 9)  Laboratories  – The certification body should have access to the laboratory services needed to support the audit program functions and document these laboratory services, including those obtained through agreements with external laboratories.
  10. 10)  Notification to FDA – The certifying body should notify FDA (a) when a food/feed safety issue is discovered by an auditor that may cause serious adverse health consequences or death to humans or animals; (b) when certification has been withdrawn from an establishment; and (c) 60 days prior to any significant change in the certification program (once program is recognized by FDA).
  11. 11)  Conflict of Interest – The certification body and its auditors should be free from conflicts of interest.  The certification body should have a committee or management structure for safeguarding impartiality.  Conflict of interest policies for a certification body and auditors acting for the certification body should be written.

A. CB Self-Assessment

Pilot Purpose - While the purpose of the pilot was for FDA to gain operational experience with third-party certification, a part of that operational experience involved the actual assessment of the CBs' abilities to meet the attributes outlined in FDA's Final Guidance for Industry - Voluntary Third-Party Certification Programs for Foods and Feeds. Therefore, the information presented on the CBs in the following sections of the report should be viewed primarily as a commentary on the kinds of challenges and issues FDA will need to address as it seeks to establish a functional third-party certification program.

Self-Assessment is a key attribute in the final guidance for Voluntary Third-Party Certification Programs for Foods and Feeds, used by a CB to assess its performance in relation to the other attributes of its certification program.  FDA's evaluation of the CBs' self assessments was a key component of the Pilot.

To help ensure completeness and consistency of information about the participating CBs' self assessments, FDA developed a series of self assessment checklists that were based on FDA Manufactured Food Regulatory Program Standards[10] and the guidance for Voluntary Third-Party Certification Programs for Foods and Feeds.  The agency held a conference call with all of the CBs in January 2009 to discuss and answer questions about the Pilot and the self assessment checklists.  FDA mailed the self assessment checklists to the Pilot CBs in February 2009, and CBs were given approximately six weeks to complete and return them to FDA.

Observations:  A review of the self assessment information provide by the CBs yielded the following observations:

  • With few exceptions, Pilot participants indicated in their self assessments that their program performed adequately against most of the attributes in the guidance.  However, FDA discovered in many cases that information in the checklist responses provided by the CBs was not adequately supported by source documents reviewed during FDA's onsite program assessment.
  • Regardless of the adequacy of the responses, not all key information necessary for assessing the adequacy of the CB's performance against the attributes could be gleaned from the self assessment checklists.  This made the onsite program assessments and associated discussions with CB personnel critical to FDA's evaluation of the CB programs.

Future Considerations:  From the review of the process for CB self-assessment the following considerations are offered:

  • Consider requiring completion of the self assessment checklist, including submission of supporting documentation, as a precondition for FDA consideration of a CB request.
  • Consider modifying the self assessment checklist to help CBs better understand the attributes and the information and documents that are needed to support the CBs' self assessments,  This may include requesting the CBs use specific terminology--established by FDA--to describe their adherence to attributes, which will assist FDA in evaluating the adequacy of the self assessments. 
  • Consider clarifying that FDA expects CBs to demonstrate adequate performance against the attributes in the guidance at the time they apply for recognition.

B. Onsite Assessment of Certification Bodies

The guidance makes clear that FDA may conduct a review of a CB's documents and records relevant to its conformance with the attributes.  The importance of the onsite assessment was underscored by deficiencies revealed during FDA's onsite review of Pilot participants' responses to the self assessment checklist.  While not explicitly stated in the Pilot announcement, FDA intended to conduct this review onsite at the CBs' international headquarters. 

FDA sent teams of at least four people to conduct the onsite assessment of each CB's program at its headquarters.  The FDA teams included individuals with previous auditing experience and technical expertise in relevant FDA regulations for aquacultured shrimp.  Each on-site program assessment was conducted over a period of three to five days.  The length of the audit varied, depending upon the size of the CB's certification program and the extent to which the CB was prepared for the assessment.  FDA's assessment of each certification program focused on reviewing records that supported the CB's self assessment.  FDA team members worked closely to maintain a consistent approach, including conducting the first assessment with all FDA assessment team members present.

Observations:  FDA's experience with the on-site assessments yielded the following observations:

  • Most of the CB certification programs were using something other than FDA's seafood HACCP regulations as their criteria for certifying aquacultured shrimp processors, e.g. BRC.[11]   Nevertheless, FDA made a policy decision to proceed with its observations of CB auditors conducting certification audits, believing the agency could still obtain useful information that would be important in designing any future third-party certification program.
  • FDA's onsite assessments were predicated on the assumption that all supporting documentation and relevant personnel would be available at the CB's international corporate headquarters.  In some cases audit information and/or analytical reports were maintained at local offices and were not available for review at the corporate headquarters.  When the relevant documentation and CB personnel were not available, FDA's ability to make a full assessment of one or more program attributes was limited.
  • Some CB records were not available in English, and not all CB personnel involved in the onsite assessments spoke English.  While interpreters were available and supplied by the State Department for assistance, there were logistical challenges in arranging for interpretation and translation services.
  • CBs were interested in (and often requested) immediate feedback from FDA during the onsite assessment, including assessment findings and suggestions on how to improve their programs.  FDA was not able to provide formal feedback during the assessment because the assessment report and conclusions had not yet been completed.
  • CBs were interested in (and often requested) training from FDA on U.S. regulations and food safety requirements.

Future Considerations:  From FDA's onsite CB attributes assessment experiences, the following considerations are offered:

  • Consider, prior to the onsite visit, requesting: 1) a dedicated point of contact to facilitate communication, 2) identification of relevant personnel that need to be available during the assessment, and 3) production of a master list of records maintained by the CB supporting its conformance with the attributes.
  • Consider devoting more time to pre-assessment communications with CBs to clarify expectations and maximize the productivity of the onsite assessments.  This is critical for CBs that conduct business and maintain records in languages other than English.
  • Consider sharing relevant findings with the CB immediately after an onsite assessment.  When such feedback cannot be immediately shared, consider communicating to the CB an expected timeline or process for when feedback could be provided.
  • Consider mechanisms to leverage external training courses for CBs, in the context of available resources and other resource commitments.

C. Overall Assessment of CBs' Ability to Meet Attributes

Based on data collected during Phase 2 of the Pilot (e.g., from review of the self assessments and the onsite visits to participating CB headquarters), FDA rated each CB on its ability to meet the attributes in FDA's guidance (see Appendix A).

Each attribute had multiple sub elements and for purposes of the pilot program each attribute was assessed one of three qualitative ratings:

Fully Met - indicating the CB fully met all the sub elements for a given attribute.

Partially Met - indicating the CB met some of the sub elements of the attribute, but not all.

Not Met - indicating the CB did not have or meet any of the sub elements satisfying the attribute.

N/A – indicating the attribute was not applicable to the CB

Table 3 – CB Program Attributes
Attributes Certification Bodies' Ability To Meet Attributes
Fully Met Partially Met Not Met N/A
Industry Relations 6 -- -- --
Resources 5 1 -- --
Laboratories 3 1 -- 2
Conflict of Interest 4 2 -- --
Elements of an Effective Audit Program 2 4 -- --
Compliance and Corrective Actions 2 4 -- --
Qualifications and Training for Auditors 1 5 -- --
Self Assessment of the Overall Certification Program -- 6 -- --
QA Program for Audits and Auditors 2 3 1 --
Authority of the Certification Body 1 4 1 --
Notification to FDA 2 2 2 --

Findings:

  • Attributes Fully or Partially Met by All CBs (see green shaded area in Table 3).  Nearly all Pilot participants fully met the attributes, where applicable, for: Industry Relations, Resources, and Laboratories.  The attribute for Conflict of Interest was also a strong area for the Pilot participants with most (4 of 6) CBs fully meeting the elements for the attribute.
  • Attributes Only Partially Met by Most CBs (see yellow shaded area in Table 3 above).  These included the attributes for Elements of an Effective Audit Program, Compliance and Corrective Actions, Qualifications and Training for Auditors, and Self Assessment of the Overall Certification Program.  Most (3 of 4) of the attributes in this category had at least one participant who fully met all the elements.  However the Self Assessment attribute was only partially met by all participants.
  • Attributes Not Met or Only Partially Met by Most CBs (see red shaded area in Table 3).  Two of the three attributes in this category: Authorities, and Notification to FDA, are tied directly to FDA requirements.
  • Attributes Not Applicable to CBs.  The attribute of Laboratories was not applicable to two CBs because they did not require laboratory testing as a condition of certification.

Observations:

  • The observed CB program did not always match the CB's self assessment.
  • The food safety scheme classifications used by some CBs differ from those of the agency. For example, areas that could potentially be considered critical by FDA were identified as non-critical by some of the CB auditors as per their schemes.
  • Although some CB programs initially appeared comparable to FDA's requirements, further review of audit reports and program implementation during the on-site audits revealed substantial differences.
  • Some participating third-party certification programs based outside the U.S. were based on European guidelines, such as BRC, as opposed to U.S. regulatory requirements.
  • Some of the attributes such as Compliance and Correction Action and Notification to FDA, appear to be more easily met when a CB is a regulatory body.  Government competent authorities were stronger in these areas than private CBs.
  • Some attributes were not directly applicable to the CB.  For example, the laboratory attribute encompasses responsibilities for both the CB and the laboratory itself, without clearly specifying the role of laboratory analyses in reaching or revising certification decisions.
  • In countries that speak a language other than English, translation and interpretation are essential to verifying factors assessed by the CB. 
  • Some of the CBs did not use risk-based criteria to determine which firms to audit or what types of products and processes to focus on during the audit. Instead, firms were chosen based on contractual agreements.
  • It became clear, during the assessment phase that some CBs' auditing personnel were not provided with any training or knowledge specific to FDA's general food, seafood, labeling, or other regulatory requirements.
  • CBs maintained records for varying lengths of time; the shorter the period, the more difficult it was for FDA to gather evidence needed to support the attribute assessment, especially in the case of previous audit reports for the attribute "Quality Assurance Program for Audits and Auditors."
  • Elements of some of the attributes, including "Qualifications and Training for Auditors" and "Industry Relations," appeared to be overly prescriptive or go beyond the scope of what a private third-party should be expected to do.

Future Considerations:

Based on FDA's experience in the Pilot, it is evident the participating CBs did not fully understand the attributes in the guidance for Voluntary Third-Party Certification Programs for Foods and Feeds.  Clarification or modification of the attributes will likely be necessary for standing up a full-scale third-party certification program.  FDA also should consider mechanisms to leverage existing external training resources, such as establishing means to communicate external training opportunities to the CBs.

Phase 3: Audit Performance of Certification Bodies:

Phase 3 of the Pilot focused on observing CB auditors conducting actual audits of shrimp processors and farms.[12]  FDA evaluated auditors using FDA's Level II Seafood Investigator Certification Audit criteria to assess the alignment of each CB's audit program with FDA's own investigator audit program.  FDA also conducted direct FDA audits of laboratories used by the CBs to support their program functions.  FDA also observed CB auditors to compare their performance against the CB's own auditing criteria.  The following timeline illustrates the key Phase 3 milestones of the Shrimp Pilot, which took place from March through September 2009.

 A timeline showing milestones in Phase 3 of the Shrimp Pilot beginning with the beginning of schedule of performance audits on March 17, 2009; the start of conducting performance audits on July 13, 2009; and the completion of performance audits on September 25, 2009.

Figure 4. Phase 3 Milestones

FDA conducted a total of 36 audits of CB auditors at processors and farms and conducted direct FDA audits of 11 laboratories used by the CBs. The audits were spread across seven foreign countries (see table below).

Table 4 – Number and Location of Audits
Country Processor Audit Farm Audit Laboratory Audit Total
China 7 -- 1 8
Ecuador 2 2 1 5
Indonesia 4 2 1 7
Nicaragua 1 1 -- 2
Singapore -- -- 1 1
Thailand 11 3 5 19
Vietnam 3 -- 2 5
Total 28 8 11 47

Logistics for Conducting CB Auditor Performance Assessments

Audit locations were chosen based on several factors including the desire to observe audits of processors in a diverse set of countries; and audits of processors, farms and laboratories involving the greatest volume of products being shipped to the United States, as well as the need to include a sample of processors, farms, and laboratories from each participating CB.  The selection of audit locations was further complicated in that FDA was limited to observing auditors only at those processors, farms, and laboratories willing to grant access to FDA.

Scheduling of audits required coordination between multiple parties and organizations including FDA, the CB, the competent authorities in the countries where the audits were being conducted, and the processors, farms, and laboratories being audited.  While FDA's preference during the Pilot was to observe actual certification audits, when schedules could not be aligned, some CBs conducted mock audits to accommodate FDA's availability.[13] 

FDA encountered a variety of challenges in planning, preparing for, traveling to and conducting audits of third-party audits for Phase 3 of the Shrimp Pilot.  These challenges included:  

  • Unpredictability of World Events - International crises in countries where audits were to be conducted complicated the planning process, with a last minute change in Indonesia due to a bombing in Jakarta and a Honduras audit being canceled due to civil unrest. The latter cancellation also affected the nearby Nicaragua audit that was planned for the same trip and audit team.
  • Adapting to CB Certification programs - Several of the CBs' certification criteria covered areas other than food safety, e.g. social responsibility.  FDA requested that auditors conduct the relevant food safety parts of their audits first to allow FDA to more efficiently evaluate the portions of the audits that had applicability to FDA-related issues.  In addition, during the audits FDA personnel attempted to minimize their disruption of the usual audit process, but it was difficult due to the need for interpretation and to ask clarifying questions.
  • Participation of FDA staff in onsite assessments and processor audits - Having the same FDA personnel participate in both the onsite program assessments and the processor audits for the same CB was preferred.  However, due to the logistical limitations, it was not always possible.

A. Assessments of Processor Audits

1. Assessment of Certification Body Auditors using FDA Level II Seafood HACCP Investigator Certification Auditor Criteria

FDA's Level II Seafood HACCP Investigator Certification audit criteria,[14] with additional key audit performance elements listed in the Voluntary Third-party Certification Programs for Foods and Feeds Guidance, were used as a baseline to evaluate the performance of CB auditors to assure that FDA's food safety requirements, e.g., processor adherence to FDA's seafood HACCP regulations, were met.

The twenty-eight observed processor audits involved 23 CB auditors, including four auditors who conducted multiple audits. CB auditors were evaluated to determine if their performance in determining a shrimp processor's compliance with the CB's certification criteria would result in compliance decisions consistent with such decisions made by an FDA Level II certified investigator.  Evaluation criteria were grouped into two categories of performance elements: "critical" and "non-critical." 

Critical Performance Elements

Critical performance elements (see Appendix B) were associated with key knowledge, skills, and abilities that, if not demonstrated by the auditor, could result in the failure of the auditor to detect the processing of potentially unsafe food.  These included the auditor's ability to evaluate the processor's state of control; recognition of processor deficiencies in identification and control of hazards, sanitation, and sanitation monitoring; adequate evaluation of the sanitary condition of the processor; recognizing, collecting, and identifying appropriate evidence; auditor's ability to explain adverse findings clearly; and the auditor's ability to adequately document observed findings.  There were 11 critical elements assessed during the processor audits.  Summary results of critical element assessments are displayed in Appendix B.

Scoring was based on three classifications: met, not met, and partially met.  A "met" indicates that the auditor fully met a given element; a "not met" means that they did not meet the element. A "partially met" score was assigned to elements where the auditor's performance could not be rated as either fully meeting or not meeting a given element.

Three of the eleven critical elements (28%) were met in 70% or more of the observed processor audits relating to auditors' (1) properly documenting and collecting adequate documents and information to support audit observations; (2) clearly explaining adverse findings; and (3) clearly documenting food safety findings in accordance with the certification program policy and providing a copy to the establishment's person in charge.  These elements are color-highlighted in green in Appendix B.   An additional three of the critical elements (28%) were not met in 70% or more of the observed processor audits, relating to auditors' (1) understanding of how to identify, evaluate, and control the food safety hazards associated with the product and process being audited; (2) recognizing deficiencies through HACCP plan and record review; and (3) recognizing, through in-plant observations, deficiencies in the identification and control of hazards.  These elements are color-highlighted in pink in Appendix B.  Of the remaining five critical elements, approximately half of the processor audits met the element and half did not.

Non critical performance elements

Non-critical performance elements evaluated knowledge, skills, and abilities that were not directly related to the auditors' ability to detect the processing of potentially unsafe food or the effectiveness of the processor's HACCP program. These include the auditor's communication skills, decision making capabilities, professional behavior, administrative skills, collection of information, and evaluation of the processor's administrative procedures.  There were 40 non-critical elements assessed during the audits (see Appendix C).

Scoring was based on the same three classifications as for critical elements: "met," "not met" and "partially met."  However, the non-critical elements also allowed for a rating of "not applicable," or NA, because the performance for that given element was not observable during the audit.

Fifty-eight percent (23 of 40) of the non-critical auditor performance elements were met during the majority of the 28 processor audits (highlighted in green in Appendix C) and only 3 (or 7.5%) of the non-critical elements were not met during the majority of the 28 audits relating to auditors' (1) inquiring if the establishment or its products were the subject of an FDA Import Alert; (2) adequately evaluating finished product labeling; and (3) evaluating conditions that could result in misbranding or economic fraud and recognizing deficiencies.  These elements are highlighted in pink in Appendix C.

For both critical and non-critical performance elements, performance among auditors of the same CB varied, with some CB's auditors exhibiting high uniformity and others exhibiting less.  In addition, since some of the performance elements are closely linked to each other, failure to meet one of the related elements increased the likelihood of not meeting the related elements.

2.  CB Protocol Implementation Assessments

The goal of these assessments was to observe how closely the CB auditors followed their own CB certification guidelines (regardless of whether or not FDA considered those guidelines as meeting FDA requirements).  These assessments were not intended to be used to make comparisons of program performance between certification bodies.  The assessments were conducted during some but not all of the audits used to assess auditors' performance with FDA Level II Seafood HACCP certified investigator criteria.

Certification scheme implementation assessments were based on the CB's audit procedures (protocols).  FDA observers compared the auditor's performance against the CB program's protocols.  Due to staff availability and translation difficulties, FDA was not able to evaluate all audits against CB protocols.

Six of the seven CB protocols observed[15] used check-listed criteria that the auditors used to record their findings and evaluate the processor's or farm's compliance with certification requirements.  These protocols included program criteria that are beyond FDA's inspectional requirements.  Of the protocols observed, one was closely aligned with FDA's inspection procedures and provided less prescriptive inspectional instructions than did the check-listed approach.  Four of the CBs used the same for-profit certification protocol involving check-listed criteria as the basis for their certification programs.

Prior to each audit, the CB auditor was counseled by FDA to fully explain each step of their inspection and their rationale for decision-making in order to accurately assess the auditor's performance against the CB's own certification protocol.  However, when FDA observers could not verify that a protocol requirement was met, either through direct observation or through verbal communication from the CB auditor, those requirements were considered as "not observed," which, for purposes of this pilot, ultimately counted as not being satisfactorily completed.

Given the differences between protocol requirements, limited numbers of observed audits, and auditors for each CB, the results summarized below are not fully quantitative. 

Of the six CBs involved in the pilot, assessment of CB auditors at processing facilities (and farms) indicated that only two of the CBs were successfully implementing their own protocols. One of these CBs used internationally recognized for-profit protocols that required completion of check-listed criteria and a ranking of processor (or farm) performance.  For this CB, FDA observed two processor audits and two farm audits.  The processor audits were performed by the same team of two auditors and the farm audits were conducted by a different team of two auditors.

The second CB's certification protocol was based on work instructions for activities to be accomplished during an inspection.  FDA observed 10 audits of processors (all within the same country), conducted by 10 different lead auditors.  The auditors for this CB displayed a consistent adherence to their protocols and were assessed as uniform and successful in implementing their CB's protocol.

Three of the remaining four CBs were assessed as not adequately implementing their CB certification protocols.  All four used check-listed criteria, with two of the CBs using an internationally recognized for-profit certification protocol.  FDA observed between one and five processor audits for each of these CBs, involving from one to three auditors per CB. The CB, for which five audits (in four countries) were assessed, involved three different auditors.  FDA team members observed variability in protocol implementation between the three auditors, with two of the three auditors failing to address a significant number of inspectional points and failing to adequately assess critical deficiencies defined in their CB certification protocol.

Assessment of the final CB, which uses check-listed criteria to determine a score that indicates the degree of compliance the processor (or farm) has with the CB certification requirements, was inconclusive.  FDA audited the CB implementation protocol at two processing facilities (and two farms).  However, the CB decided to abbreviate its protocol during these audits at the request of the processors, since the CB charges for every audit.  As a result, the CB auditors conducted only limited audits at these facilities that did not cover the entire scope of the CB's full protocol. FDA observers were unable to determine if items missed in the protocol were intentionally left out or resulted from auditors' unfamiliarity with the protocol.  In addition, the auditors for this CB are subcontractors who are expected to bring their own equipment necessary to accomplish their audit tasks.  FDA audit team members observed that these auditors did not bring their own equipment, but rather used equipment provided by the processor or asked processor employees to conduct tests for them.

B. Assessments of Farm Audits

FDA teams conducted eight farm audit assessments.  The farm auditor assessments included elements FDA considers necessary to ensure food safety and sanitary conditions. The criteria for assessing the performance of auditors during the farm audits was specifically developed for the Pilot and took into consideration the auditor's recognition and identification of food safety and sanitation controls on aquaculture farms.  Each CB was provided with a copy of the assessment criteria, in English, prior to the beginning of FDA's onsite audits, along with instructions for assessing performance based on FDA's requirements.

Critical Elements

There were five critical farm audit performance criteria (see Appendix D) covering the areas of:  processor verification procedures; aquaculture drug use; farm examination; and evidence.

There were few opportunities to observe the processor verification procedure (i.e., auditor recognition and documenting of inadequacies in the processor's farm inspection procedures, or written farm inspection report or corrective actions taken).  However, in all three observed opportunities, auditors failed to recognize and document inadequacies in the processor's written inspection report or corrective actions taken in response to that report (see element highlighted in pink in Appendix D).  CB auditors did better when verifying the processor's farm inspection procedures (elements highlighted in green in Appendix D).

Non-critical Elements

There were 15 non-critical farm audit performance criteria in the areas of: processor verification procedures; aquaculture drug use; farm examination, documentation review; and aquaculture feeds (See Appendix E).  Auditors performed well with regards to identifying unapproved drugs, assessing the proper use of approved drugs, and inquiring about the source of shrimp larvae.  However, auditors in only two of eight audits successfully demonstrated adequate knowledge of aquaculture drugs approved for use by FDA.

Observations from Processor and Farm Audits

  • Auditor Performance Observations:
    • –  Wide variation in CB auditor performance across CBs
    • –  Less variation, although in some cases still significant, in auditor performance within a single CB
    • –  Strong performance in assessment of GMPs by auditors
    • –  Significant deficiencies in assessment of seafood HACCP programs, with the exception of one CB
  • Audits were likely impacted by the presence of FDA observers.  It was not possible to ascertain the level of this impact and whether FDA's presence had an effect that limited FDA's ability to observe (e.g., artificial environment) the audit results.  In some cases, CB representatives, who attended the audits in addition to the auditors, appeared to influence or coach the CB auditors during the audit, making it difficult for the FDA team to objectively assess the performance of the individual auditor.  In addition, the presence of other individuals including competent authorities and interpreters may have also influenced the behavior of both the CB auditors and the processors and farms being audited.
  • Language barriers created both technical and verification issues.  Some translators had more facility with the English language than others and the translator equipment did not always perform optimally. In addition, in at least one instance, a firm directed the translator not to translate anything firm officials said, but to only translate what the CB auditor was saying.  Nonetheless, the interpreter in this instance indicated it was FDA's request to have all conversations between relevant parties translated during the audit and review of documents.
  • Certification body auditors requested feedback on their performance during the audit and expressed a desire for training on U.S. regulations and FDA requirements.  As with program assessments, individual auditors requested immediate performance feedback during the certification audits.  A number of auditors also sought to engage the FDA observers in discussion of food safety questions.
  • Cause of inadequate audit practices difficult to determine.  When FDA observed audit practices that were less than acceptable it was not clear whether such practices resulted from auditor deficiencies or deficiencies in the CB certification requirements, or both.

Future Considerations:

Consider requiring, as a condition for accreditation, that CBs maintain agreements with establishments they certify to allow FDA to monitor or otherwise participate in certification audits as necessary.

C. Assessment of Laboratory Audits

A total of 11 laboratory audits were conducted by FDA.  The attribute for laboratories in the Guidance for Voluntary Third-Party Certification Programs for Foods and Feeds notes that a CB should have access to the laboratory services needed to support its audit program functions and document these laboratory services.  The attribute also notes that the laboratories themselves should be capable of analyzing a variety of appropriate samples, using widely recognized methods, for assessing the compliance of establishments with applicable certification criteria, including product and environmental samples.  The guidance states that laboratories should conform to the ISO/IEC 17025:2005 standard "General requirements for the competence of testing and calibration laboratories."    

Laboratories were audited using the draft Private Laboratory Quality System Evaluation Guide[16] (referred to as the "checklist" or "guide").  The guide was for FDA use and was not shared with the laboratories prior to the audits.  The laboratories were assessed on adherence to their internal procedures, as well as the information gathered when completing the checklist. Two to three FDA analysts conducted each audit with both microbiology and chemistry expertise always represented.  On average, each audit was conducted over two days. 

CB laboratories had well-documented quality systems as required by ISO 17025.[17] Many laboratories indicated that they had experience with audits conducted by ISO accreditors, as well as other governments (i.e., the EU), customers, and internal auditors.  Nevertheless, some of the laboratory systems did not fully meet the FDA laboratory checklist recommendations.

Several CB laboratories mentioned that it is very difficult to stay up to date on FDA methodology due to the format and organization of the FDA website.  As a way to simplify sharing of information, CBs suggested that FDA use the Association of Analytical Chemists (AOAC) practice of emailing stakeholders/subscribers when methods are updated.

Observations:

Addressing Nonconformities: In general, the audited laboratories were not proactive in preventing nonconformities. They did not routinely conduct a root-cause analysis for identified deficiencies or test deviations.  

Training and Personnel: FDA observed several instances where laboratory personnel did not fully demonstrate their understanding of why a certain test was being performed and the significance of the results. In addition, several laboratories did not have a mechanism in place for performance improvement if staff performance is deficient. 

Chain of Custody and Integrity of Samples:  Not all laboratories had a robust traceability system to ensure adequate chain of custody. There were also several examples where samples were collected by the clients who requested the sample, which poses a potential conflict of interest.

Impartiality of Sample Testing: Several laboratories exhibited shortcomings in maintaining the confidentiality of samples; blind testing was not always observed.  Another practice observed was the retesting policy where a laboratory routinely retested samples that were out of specification and accepted the results that were within specification without retesting the whole batch.

Future considerations:

  • Consider requiring more technical details in the CB application package before visiting the laboratory.
  • Consider requiring laboratories to take a more proactive approach for preventing non-conformities, to include routinely conducting root-cause analyses for any identified deficiencies or test deviations.  Such analyses would allow the laboratory to assess whether a corrective action was effective and would also form the basis to prevent nonconformities in the future. 
  • Improve the FDA laboratory methodology section on FDA's website and provide the link to all CB laboratories.
  • Consider sending out email updates to laboratories on FDA methodologies as is done by AOAC.
  • Consider developing a standardized checklist that facilitates the assessment of both a CB laboratory program and its ongoing operation for future audits.
  • Consider allotting more time (more than two days) for audits of larger laboratories.

Pilot Administration and Logistics

This section of the report provides a discussion of the overarching administrative and logistical issues encountered in the conduct of the Pilot.  These are issues that should be addressed in order to operationalize a third-party certification program at FDA.  At the outset, it is important to note that existing FDA infrastructure was taxed during the Pilot, indicating that an operating program in the future would need additional resources to be successful, as well as a central coordinating point within the Agency.

Administrative Issues

The cross-organizational nature of the Shrimp Pilot argues for designation of an overall program lead who has sufficient authority to make decisions as well as coordinate and implement the day-to-day operations of a third-party certification program. 

The Shrimp Pilot required a large amount of overseas travel (see table below), and coordination among multiple stakeholders demanded significant time and resources during the Pilot.  Several FDA components contributed to the effort along with other federal agencies, e.g., the State Department.  Challenges included internal FDA logistics (travel planning and preparation, in country support), external logistics (coordinating with CBs and firms, in-country travel), interpreter services, and international relations (diplomacy and coordination with the relevant competent authorities).

Assessment and Audit Costs – Time and Travel

FDA was required to travel extensively to complete the program assessments and audits necessary to conduct the Pilot. The following table illustrates the number of onsite assessments and audits conducted.

Table 5 – Location and Number of CB Program Assessments and Audits
Locations of Program Assessments and Audits
Country Program
Assessments
Processor
Audits
Farm
Audits
Laboratory
Audits
Total
Argentina 1 -- -- -- 1
China -- 7 -- 1 8
Ecuador -- 2 2 1 5
England 1 -- -- -- 1
France 1 -- -- -- 1
Indonesia -- 4 2 1 7
Nicaragua -- 1 1 -- 2
Singapore -- -- -- 1 1
Thailand 1 11 3 5 20
United States 2 -- -- -- 2
Vietnam -- 3 -- 2 5

Information Technology Issues

The Shrimp Pilot relied on information technology (IT) to facilitate communication between FDA and the CBs (blackberries, laptop computers), share information between the various workgroups (email, e-Rooms, and online repositories), and to capture information for internal reporting systems.  Although existing IT communication tools, e.g., email and blackberries, performed adequately during the Pilot, current IT systems and databases were not designed to accommodate third-party certification audits.  IT needs will require more in-depth evaluation, updating, and the potential development of new systems and databases should FDA operationalize a third-party certification program. 

Communications Issues

Clear and consistent communication between internal and external stakeholders was an essential component of every aspect of the Shrimp Pilot.  The international aspect of the Pilot highlighted the importance of communication not only between FDA and the CBs, but also other stakeholders such as foreign governments.  In addition, FDA discovered it should have engaged the competent authorities (CAs) of host countries earlier in the process to give them more time to prepare and potentially observe the audit process. 

When communication faltered or was insufficient, delays and misunderstandings ensued.  When communication was clear and consistent, it improved both the quality and quantity of information that FDA gathered during the Shrimp Pilot.  Successful communication also helped prevent duplicative work, which was vital for a large project that operated with limited resources.

Language barriers and different operating models and paradigms (i.e., industry vs. regulatory) made understanding between FDA and the CBs challenging and clear communication even more critical.  At the beginning of the Pilot, both FDA and the CBs had multiple points of contact, duplicating work and possibly sending inconsistent messages.

Interpreters were essential during both on-site CB assessments and processor/farm/laboratory audits.  FDA auditors often had to ask clarifying questions, while also understanding what was being discussed between the CB auditor and the firm.  Interpreters ensured that the CB and FDA were able to communicate clearly, which resulted in a more accurate audit.  It should be noted that the number of interpreters needed for a full-scale third-party certification program are likely to be substantial.  While FDA relied primarily on the State Department to provide interpreter services during the Pilot, this arrangement will need further evaluation if the Agency operationalizes a third-party program.

Training Issues

During the Pilot, FDA provided training for FDA personnel involved in the on-site assessments and performance audits.  Training topics included shrimp farming concerns, drug residues, and laboratory issues, as well as audit team roles, procedures, protocols, and reporting.

Looking forward, if FDA seeks to operationalize a third-party certification program, establishing robust formal training for Agency personnel involved in on-site program assessments and performance audits of CB auditors and supporting laboratories will be critical to ensure consistency in application of FDA criteria for these functions.

A common area of feedback from the CBs that participated in the Pilot was the desire for training on FDA's regulations in order to better design their certification programs to meet the Agency's requirements.  During the Pilot FDA learned that many of the participating CBs had an expectation that FDA would provide such training as a benefit of participating in the program.  In addition, based on FDA's experience in the Pilot, it was evident that the participating CBs did not fully understand the attributes in the guidance for Voluntary Third-Party Certification Programs for Foods and Feeds.  More CB training on and clarification or modification of the attributes will likely be necessary in order to stand up a full-scale third-party certification program.

Application of Pilot Findings to Future Use of Third-party Certification (TPC) Programs

The Shrimp Pilot was a learning process for FDA, the CBs, and the aquaculture industry.  While the Agency had some idea of the types of issues it might encounter, it was difficult to anticipate the magnitude of some of the challenges encountered in actually implementing the Pilot.  Based upon FDA's experiences under this Pilot, the Agency will need to more fully explore communication, logistic, administration, and training options for conducting future third-party programs.

Key considerations for designing and standing up a full-scale third-party certification program include the following:

  • FDA should consider structuring the program to provide a well-defined point of coordination and decision-making within the Agency.
  • FDA will need significant logistical and resource support to establish and maintain a third-party certification program.  Direct accreditation by FDA of third-party certification bodies would be particularly resource intensive.
  • FDA should enhance existing and create new IT data systems to capture and report on results of assessments and audits.
  • FDA should be clearer about its expectations for application to a third-party certification program. FDA should make its application process more clear and transparent with more opportunities for early interaction with FDA.
  • FDA should create standardized procedures and forms to use in evaluating participants in a third-party certification program and train FDA personnel in these standards.

Conclusion

The Shrimp Pilot was intended to provide FDA with technical and operational information to assist the Agency in determining infrastructure needs, as well as assessing the process for evaluating third-party certification programs.  The pilot data indicate that having the appropriate infrastructure in place at FDA will be critical to the success of any full-scale program.  The experience also raises a number of other issues for FDA to consider.  In conclusion, FDA views the Shrimp Pilot as a useful and data-rich step toward establishing a third-party certification program that will provide FDA with increased confidence that food or feed from foreign producers meets U.S. safety standards.

****

Glossary of Terms as Used in this Report

Accreditation means an attestation related to a CB (but not by the CB itself) conveying formal demonstration of its competence to carry out specific certification tasks.

Accreditation body is an authoritative body that performs accreditation.

Attributes are characteristics of a CB and its certification program that provide a reliable reflection that foods from an establishment certified by the CB meet applicable FDA requirements, as well as other certification criteria.

Audit is the systematic and functionally independent examination of a product, process, and firm, including records and laboratory testing, as appropriate, to determine a firm's conformance with certification criteria. Audits may include a range of activities, such as onsite examinations of establishments, review of records, review of quality assurance systems, and examination or laboratory testing of product samples.

Auditor is a person acting for the CB, the FDA, or other entity, who assesses and makes a determination of the degree to which certification criteria of a recognized scheme have been met by a firm for a particular product type.

Certification is the procedure by which a CB provides assurance that the firm conforms to certification criteria. Certification should be granted for particular product types produced, manufactured, processed, packed, or held by the firm. Certification should be, as appropriate, based on a range of audit activities.

Certification body (CB) is a third-party organization that operates a certification program. A CB could be a Federal, State, local or foreign government agency, as well as a non-government entity that is independent of the businesses it certifies and free from conflicts of interest.

Certification criteria are criteria used by the CB during an audit to determine whether a firm should receive certification. Certification criteria for the purpose of this document should, at a minimum, include applicable FDA requirements.

Certification program is system that verifies, through audits, conformance with certification criteria.

Certification program assessment is a systematic examination by FDA to assess a CB's conformance with the attributes.

Firm is a site-specific domestic or foreign facility that produces, manufactures, processes, packs, or holds food for use, consumption, or further processing in the United States.

FDA authorized inspection is the examination of a product, process, or firm, including records and laboratory testing, by FDA (or another governmental entity acting under FDA's authority, such as a State regulatory authority, with which FDA has a contract, partnership arrangement, or memorandum of understanding (MOU)) for the purpose of conducting inspections to establish the firm's compliance status with FDA requirements.

Third-party is an organization other than the firm or FDA (or another governmental entity acting under FDA's authority, such as a State regulatory authority, with which FDA has a contract, partnership arrangement, or MOU for the purpose of conducting inspections that pertain to the firm's compliance status with FDA requirement). A third-party could be a Federal, State, local or foreign government authority that is not conducting inspections under FDA's authority, as well as a private entity.

Acronyms Used in this Report

Acronyms Used in this Report

AOA C  Association of Analytical Communities

BRC   British Retail Consortium

CA   Competent Authority

CB   Certification Body

EU   European Union

FAO   Food and Agricultural Organization

FDA   Food and Drug Administration

FR   Federal Register

FSWG   Food Safety Working Group

GA   Government Accountability Office

GMPs   Good Manufacturing Practices

HACCP   Hazard Analysis and Critical Control Point

IEC   International Electrotechnical Commission

ILAC   International Laboratory Accreditation Cooperation

IOM   Institutes of Medicine

ISO   International Organization for Standardization

IT   Information Technology

NRC   National Research Council

TPC   Third-Party Certification

 

Appendices


Appendix A – CB Attributes and Sub Attributes


CB Attributes – Sub Attributes

  1. 1 – Authorities of the Certification Body
    • Perform audit activities
    • Examine, gather, and copy records
    • Authority to collect and analyze  samples
    • Authority to certify, recertify, or withdraw certification
    • Authority to report findings to FDA
    • Adequate controls for meeting FDA laws and requirements
  2. 2 – Qualifications and Training for Auditors
    • Education
    • Training - Coursework
    • Training – Field
    • Training - Specialized
    • Continuing Education
  3. 3 – Elements of an Effective Audit Program
    • Risk based
    • Written policies and procedures
    • Access to establishment without undue delays or limits
    • Audit complaint process
    • Audit documentation
  4. 4 – Quality Assurance Program for Audits and Auditors
    • Initial assessment program
    • Ongoing assessment program
    • Quality Assurance Program (QAP)
    • Audit report evaluation
    • Sample report evaluation
    • Auditor performance
    • Certification Program Performance
  5. 5 – Compliance and Corrective Actions
    • Tracking system
    • Written procedures and timelines
    • Annual review
  6. 6 – Industry Relations
  7. 7 – Resources
    • Staffing
    • Equipment
    • Funding
  8. 8 – Self Assessment of the Overall Certification Program
    • Initial self-assessment
    • Self-assessment of program every 36 months
    • Self-assessment worksheets
    • Corrective action plans
    • 3rd party verification
    • Maintain supporting records
    • FDA notification
  9. 9 – Laboratories
    • CB has access to laboratory with necessary capabilities
    • CB maintains records of services
    • Contract or written agreement with laboratory
    • Laboratories are signatory to ILAC Recognition Agreement
    • Laboratories have ISO 120/IEC 17011 Accreditation
    • Laboratories implement and conform with ISO/IEC 17025 criteria
    • Laboratories use appropriate laboratory methods
    • Laboratories implement written QAP program that contains recommended controls
  10. 10 –Notification to FDA
    • Written agreements with establishments
    • Up-to-date written notification procedures that is signed by CB Officials
    • Periodic testing of system
  11. 11 – Conflict of Interest


Appendix B – Critical Performance Elements re: Processor Audits


PROCESSOR AUDIT ASSESSMENT -- CRITICAL ELEMENTS
(28 Audits)
Assessment Area Performance Elements Met Partially
Met
Not Met
# % # % # %
Ability to Evaluate Firm's State of Control 1. Did the auditor demonstrate an understanding of how to identify, evaluate and control the food safety hazards associated with the product and process being audited? 4 14% 2 7% 22 79%
2. Did the auditor demonstrate an understanding of how to control and monitor sanitary practices? 12 43% 1 4% 15 54%
Performance of the Audit 1. Did the auditor adequately assess the firm's implementation of their HACCP plan (or necessary process controls in the absence of a HACCP plan) and sanitation monitoring? 12 43% 2 7% 14 50%
2. Did the auditor recognize, through in-plant observations, deficiencies in the identification and control of hazards? 4 14% 2 7% 22 79%
3. Did the auditor recognize deficiencies in the identification and controls of food safety hazards through HACCP plan and record review?  4 14% 1 4% 23 82%
4. Did the auditor recognize deficiencies in sanitation and sanitation monitoring through in-plant observations? 12 43% -- -- 16 57%
5. Did the auditor recognize, through records reviews, deficiencies in sanitation and sanitation monitoring? 13 46% -- -- 15 54%
6. Did the auditor adequately evaluate the sanitary condition (based on 21 CFR 110) of the facility? 13 46% 2 7% 13 46%
Recognize, Collect and Identify Appropriate Evidence 1. Did the auditor properly document, collect and document adequate information and documentation to support audit observations and the final certification decision? 20 71% 1 4% 7 25%
Inspection Process 1.  Did the auditor explain adverse findings clearly during the exit interview? 28 100% -- -- -- --
Observations 1. Did the auditor document findings relating to food safety accurately, clearly, and concisely and in accordance with certification program policy and provide a copy of this document to the establishment's person in charge? 26 93% -- -- 2 7%


Appendix C – Non-critical Performance Elements


PROCESSOR AUDIT ASSESSMENT – NON-CRITICAL ELEMENTS
(28 Audits)
Assessment Area Performance Elements Met Partially
Met
Not Met N/A
Background 1.  Did the auditor inquire whether the firm has been audited by another third-party recognized by FDA and request and review any resulting reports? 13 -- 15 --
2.  Did the auditor inquire whether the firm has been inspected by FDA or a government agency other than FDA, request and review any resulting reports? 15 -- 13 --
3.  Did the auditor inquire if the firm had received a warning letter from FDA and, if so, request and review the warning letter? 12 -- 15 1
4.  Did the auditor inquire if the establishment or its products are subject to an FDA Import Alert? 11 -- 17 --
5.  Did the auditor inquire if the reported deficiencies had been successfully corrected to the satisfaction of the inspecting agency and review any written responses? 19 2 7 --
6.  Did the auditor inquire if the establishment or any of its officers or employees are being prosecuted or have been convicted of a crime relating to FDA regulatory requirements and, if so, inquire what crime was committed? -- -- -- 28
7.  Did the auditor inquire if the establishment had made any significant changes to the safety systems the establishment had in place during the previous audit or had occurrences that may affect product safety or certification? 26 -- 2 --
8.  Did the auditor verify the pertinent establishment background information? 25 -- 3 --
Compliance Assessment 1.  Did the auditor use or have access to the most current applicable resource documents such as program/ product specific guidance; CB protocols, firm/ farm's operation protocols, etc? 25 -- 3 --
2.  Did the auditor demonstrate that he/ she conducted preparatory reviews of the previous audit reports, consumer complaints, and other pertinent background documents? 23 -- 5 --
3.  Was the auditor able to distinguish between significant and insignificant observations relating to food safety and isolated incidents versus trends? 16 -- 12 --
4.  Did the auditor select an appropriate product for audit coverage, based upon "risk"? 24 -- 4 --
5.   Did the auditor deal effectively with obstacles to the audit or adversarial situations, if encountered? 10 -- 3 15
6.   Did the auditor verify that corrections were made in response to deficiencies or observations made during the previous audit(s)? 19 -- 6 3
7.   Did the auditor have the necessary equipment and forms needed to conduct to the audit and collect samples? 20 -- 8 --
8.  Did the auditor prioritize the audit focus and allocate their time appropriately, based upon risk? 15 -- 12 1
9.  Did the auditor adequately evaluate finished product labeling? 7 -- 20 1
10.  Did the auditor adequately evaluate conditions that could result in the food consisting, in whole or in part, of filthy, putrid, or decomposed (non-safety)? 20 -- 8 --
11.  Did the auditor adequately evaluate conditions that could result in the food consisting in whole or in part of an animal which was diseased of had died otherwise than by slaughter? -- -- -- 28
12.  Did the auditor adequately determine if appropriate Standards of Identity were met? 6 -- 7 15
13.  Did the auditor adequately evaluate food and color additives (both direct and indirect) and recognize deficiencies? 12 -- 16 --
14.  Did the auditor adequately evaluate conditions that could result in misbranding or economic fraud and recognize deficiencies? 9 -- 19 --
15.  Did the auditor seek assistance for undefined policy areas and complex technical issues beyond his or her ability at this level? 13 -- 6 9
Evidence 1. If applicable, did the auditor collect samples according to written protocols using sampling techniques that prevent contamination of the product and ensure that a representative sample is collected? 2 -- -- 26
2. Did the auditor display initiative in gathering documentation/ evidence not readily available? 16 -- 8 4
Verbal Communication 1. Did the auditor explain findings clearly and adequately throughout the audit? -- -- -- 28
2. Did the auditor use suitable interviewing techniques? 27 -- 1 --
3. Did the auditor answer questions and provide information in an appropriate manner? 28 -- -- --
4. Did the auditor ask follow-up questions to get the necessary information? 21 -- 7 --
5. Was the auditor's request for information clear and understandable? 27 -- 1 --
6. Did the auditor practice effective listening techniques? 28 -- -- --
7. Did the auditor refrain from relating confidential information? 27 -- -- 1
Professionalism 1. Did the auditor behave professionally during the audit?  27 -- 1 --
Additional Audit Attributes 1. Did the auditor request and review internal establishment self-audits and quality assurance reports? 25 -- 3 --
2. Did the auditor evaluate the establishment's method of resolving problems identified through inspections, e.g. self-audits or external audits. 20 -- 7 1
3. Did the auditor request and review appropriate records and procedures relating to food safety for the establishment's operations and effectively apply the information obtained from this review during the audit?   For example: 17 1 10 --
4. Did the auditor request and review consumer complaint files? 22 -- 6 --
5. Did the auditor inquire if the establishment used a coding system to identify production runs or other traceability factors?  If so, did the auditor obtain a description of how to interpret the code? 26 -- 2   --
6. Did the auditor inquire if the establishment had recall procedures in place and adequately evaluate the program, its records, and its controls?  23 -- 5 --
7. Did the auditor determine if the firm has recalled any product since the prior inspection and if the recall was successfully concluded? 22 -- 5 1

 

Appendix D – Critical Farm Audit Performance Elements


FARM AUDIT ASSESSMENT -- CRITICAL ELEMENTS
(8 Audits)
Assessment Area Performance Elements Met Partially
Met
Not Met N/A
Processor Verification Procedures 1.  Did the auditor recognize and document inadequacies in the processor's written inspection report or corrective actions taken in response to that report? -- -- 3 5
2.  Did the auditor recognize and document inadequacies in the processor's farm inspection procedures? 2 -- 1 5
Aquaculture Drug Use 1.  Did the auditor adequately evaluate the farm's controls for aquaculture drugs, the use of aquaculture drugs, and, if appropriate, control of treated stock? 8 -- -- --
Farm Examination 1.  Did the auditor adequately inspect the farm and recognize unsanitary conditions and/or inadequacies in the control and use of the aquaculture drugs, pesticides, or chemicals on the farm? 6 -- 2 --
Evidence 1. Did the auditor adequately evaluate farm records? Did the auditor recognize and document inappropriate drug usage and unsanitary conditions based on that review? 6 -- 2 --


Appendix E – Non-critical Farm Audit Performance Elements


FARM AUDIT ASSESSMENT – NON-CRITICAL ELEMENTS
(8 Audits)
Assessment Area Performance Elements Met Partially
Met
Not Met N/A
Processor Verification Procedures 1.  Did the auditor review processor's HACCP verification reports prior to the inspection of the farm and adequately apply that knowledge during their own inspection? 2 -- 1 5
2.  Did the auditor inquire if or what corrective actions were taken by the farm in response to any deficiencies noted by the processor? 2 -- 1 5
Aquaculture Drug Use 1.  Approved drugs - Did the auditor demonstrate knowledge of what drugs and chemicals are approved for use in aquaculture by FDA? 2 -- 6 --
2.   Unapproved drugs - Did the auditor demonstrate knowledge of what drugs/ chemicals/ pesticides, etc. are prohibited for use on aquaculture farms by the local government? Did the auditor determine if the farmer is aware of these prohibitions? 8 -- -- --
3.   Usage - Did the auditor review documents, including labels, which provided instructions on the proper use and limits of the drugs? 4 -- -- 4
4. Testing notification - Did the auditor inquire where or how the farmer obtained either their eggs or fry the farm raised? 8 -- -- --
Farm Examination 1. Did the auditor adequately inspect the farm storage areas or potential storage areas and review feed and medication labels or instructions for use on drugs, chemicals, or pesticides held on the farm? 7 -- 1 --
2.  Did the auditor adequately inspect growing conditions? 4 1 3 --
3.  Did the auditor utilize whether the farm makes its own ice? How is the water controlled? Was the ice maker reviewed? 6 -- 2 --
Documentation Review 1. Did the auditor ask about usages and review documentation? 8 -- -- --
2. Did the auditor ask about stock outbreaks and review treatment records? 7 1 -- --
3. Did the auditor adequately determine if the farm had protocols for addressing outbreaks? 4 1 3 --
4. Did the auditor inquire if other agencies or companies inspected the farm (regulatory/ other third-party / processor verification inspections)? 5 -- 3 --
5.  Did the auditor determine if the water and soil have been tested or are routinely tested for chemical residue (pesticides, PCBs, drug residue)? 5 -- 3 --
Aquaculture Feeds 1. Did the auditor inquire about the type and source of the aquaculture feeds used on the farm? 7 -- 1 --

 


[1] General Accounting Office, Food Safety: FDA's Imported Seafood Safety Program Shows Some Progress, but Further Improvements Are Needed (GAO-04-246) (Jan. 2004), available at http://www.gao.gov/new.items/d04246.pdf (noting that FDA would need to review its legal authorities in this area because, unlike devices, FDA has no statutory mandate to conduct a third-party inspection program for food). 

[2] Enhancing Food Safety: The Role of the Food and Drug Administration (National Academies Press, June 2010) available at: http://www.iom.edu/Reports/2010/Enhancing-Food-Safety-The-Role-of-the-Food-and-Drug-Administration.aspx

[3] http://www.regulations.gov/search/Regs/contentStreamer?objectId=0900006480427502&disposition=attachment&contentType=html 

[4] http://edocket.access.gpo.gov/2008/pdf/E8-15715.pdf)

[5] See final guidance posted at http://www.fda.gov/RegulatoryInformation/Guidances/ucm125431.htm

[6] 21 C.F.R. Part 123.  http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=123&showFR=1

[7] FAO CWP Handbook of Fishery Statistical Standards, Section J: AQUACULTURE http://www.fao.org/fishery/cwp/handbook/J/en

[8] FDA set out the attributes needed for a certification program in its final Guidance for Industry – Voluntary Third-Party Certification programs for Foods and Feeds.  The attributes are discussed more fully later in this report [http://www.fda.gov/regulatoryinformation/guidances/ucm125431.htm].

[9] The British Retail Consortium (BRC) global standard for food safety; see: http://www.brcglobalstandards.com/standards/food/

[10] These standards were developed as a guide to design and manage a high-quality manufactured food regulatory program and as a tool to evaluate program performance.

[11] See footnote 6.

[12] FDA's Seafood HACCP Regulation places responsibility for controlling the use of drugs on aquaculture farms with the primary processor.  Farm audits in the Pilot were observed to determine auditor's performance in assessing the adequacy of the processors' HACCP plan (which must be reassessed annually to ensure the adequacy of the hazard analysis and the whether the controls effectively control the identified hazards).

[13] Mock audits are audits that are outside of the normal certification or recertification schedule between the firm and the Certification Body (often once or twice a year). They were conducted for the benefit of FDA to observe the audit process; all aspects of the mock audits were the same as a real audit with the exception of the time since the last audit and the audit findings not being used for certification/recertification decisions.

[14] The Level II Seafood HACCP Investigator Certification audit criteria define performance assessing an FDA investigator's ability to  identify 1) food safety hazards, 2) the controls necessary to prevent or reduce the chance of those hazards occurring, 3) effective implementation of a seafood HACCP program that meets FDA's requirements, and 4) compliance with other FDA regulations.

[15] While only six Certification Bodies participated in the pilot, one of them used two internationally recognized for-profit certification schemes – one for processing facilities and one for evaluation and certification of farms.

[16] The draft guide was developed specifically to support the Shrimp Pilot program and served as a practical field tool for FDA to assess the laboratories that support certification bodies.  The tool may have future utility in assessing laboratory performance in other venues, after further revision and updating based on lessons learned from the Shrimp Pilot.

[17] ISO/IEC 17025 is the global quality standard for testing and calibration laboratories. The current release was published in 2005.  There are two main clauses in ISO/IEC 17025 – Management Requirements and Technical Requirements. Management requirements are related to the operation and effectiveness of the quality management system within the laboratory, Technical requirements address the competence of staff; testing methodology; equipment and quality; and reporting of test and calibration results.