| Comment Record |
|
Commentor |
Ms. Kate Townsend |
Date/Time |
2001-12-24 02:29:37 |
|
Organization |
Taratec Development Corporation |
|
Category |
Company |
| Comments for FDA General |
| Questions |
|
1. General Comments
|
December 23, 2001
Re: Docket No. 00D-1538, Draft Guidance for Industry; 21 CFR Part 11; Electronic Records; Electronic Signatures, Validation
Taratec Development Corporation welcomes the opportunity to comment on the FDA document titled “Guidance for Industry 21 CFR Part 11; Electronic Records; Electronic Signatures, Validation”.
We support FDA's intention to provide guidance on the requirements of Part 11 of Title 21 of the Code of Federal Regulations; Electronic Records; Electronic Signatures to enable consistency of interpretation across industry and the agency.
We include in this document our comments and observations based upon our experience of developing and validating computerized systems. We have made some general comments followed by more specific comments related to the text of the draft guidance.
General Comments
1. Many of the sections within the guidance are very useful and address some often “misunderstood” areas relating to computer systems validation. For experienced validation professionals, it is obvious that this list is not exhaustive and other areas of the validation lifecycle need to be considered. However, to less experienced readers, the statements may appear disjointed with no overall explanation of how one principle relates to the other or that some important aspects of validation are not addressed at all, for example, user training and SOPs. We recommend the addition of a section which provides a high level description of the computer systems validation process and hence presents a context for the principles described in the guidance document. An alternative approach would be to specifically reference the CDRH guidance, General Principles of Software Validation, Guidance for Industry and for FDA Staff which provides this context. Although this document is listed along with many others in the reference section, we feel that the addition of an overview section would greatly add to the usefulness of the guidance. It would also avoid further confusion since the terminology used in the referenced texts tends to vary widely from one document to the next.
2. We suggest re-ordering the sections to present the general concepts first and then order the sections according to the validation lifecycle. For example, move the section on Extent of Validation to 5.1, followed by Independence of Review, System Requirements Specifications, etc.
Specific Comments
Section 2.2
“and,” after the third bullet is not required
Section 3 Definitions and Terminology
All the terms used in the guidance should be included in Guidance for Industry, 21 CFR Part 11; Electronic records; Electronic Signatures, Glossary of Terms. Terms that are not defined include system requirements specifications, design requirements and specifications, dynamic testing, simulation tests, and static verification.
Section 4 Regulatory Requirements; What does Part 11 Require?
Part of the requirement in 11.10(a) that states “…the ability to discern invalid or altered records” is often misunderstood by industry. Discerning altered records is fairly straightforward as it relates to audit trails recording changes to the records. Discerning invalid records requires clarification and guidance from the Agency. Based on our understanding, this statement requires computer systems to identify invalid records wherever possible and flag them either at the time of data entry or during record verification. For example, if the system is prompting a user to enter a value between 1 and 10, it should flag if a value of 11 is entered. If only a value of True or False should be accepted, the system should prevent any other entries. These “rules” should be tested during validation of the system. However, it must be recognized that in some instances, invalid records cannot be identified, for example, if the system is accepting a string of text, there is often no way to determine if the text is valid or not. Clarification of the requirement to discern invalid records should be included in the guidance.
Section 5. Key Principles
As stated under General Comments, it would be beneficial if the introduction to this section stated more clearly that this is not an exhaustive list.
Section 5.1 System Requirements Specifications
1. In system development methodologies, there are different levels of requirements documents. The highest level is the end user requirements which is then translated and expanded by developers into design requirements. Section 5.1 does not make a clear distinction between the different types of requirements which could lead the reader into thinking that end users need to specify scanner resolutions and scanning rates in end user requirements. We would like to see section 5.1 clarify that end user requirements should specify the business requirement, e.g., that the accuracy and integrity of the paper record be retained during the scanning process. Subsequent technical requirements would then take this requirement and specify the appropriate scanning specifications.
2. We strongly support the statements that end user requirements must be established and documented. However, the statement “…traceable to system design requirements and specifications” appears to contradict section 6.1 that addresses Commercial, Off-The-Shelf Software. As stated in 6.1, for COTS the end user may not have access to design documentation. We recommend that a qualifier be added to the statement in 5.1 to clarify this situation. We recommend adding an additional statement that “at a minimum user requirements must be traceable to functional tests that show that the requirement has been met by the system”.
3. Much of the second part of paragraph 1 in section 5.1 appears to be re-stating the regulation. We question the value of including this in the guidance.
Section 5.2 Documentation of Validation Activity
1. Section 5.2 includes a strong emphasis on testing to the exclusion of other validation activities such as user training and the development of SOPs to ensure the ongoing maintenance of the system in a validated state. We would like to see this concept emphasized in the description of the Validation Plan and the Validation Report (see below).
Section 5.2.1 Validation Plan
1. The term “schedule of validation activities” implies that a timeline should be included in the validation plan. Although we agree that the validation activities must be listed and described along with the sequence of events and any dependencies, we do not agree that the validation plan needs to include a timeline. We request that “schedule of validation activities, and tasks to be performed” be changed to “a description of validation activities and tasks to be performed including the sequence of events and any dependencies.”
2. We suggest adding that the Validation Plan should include activities to ensure that the system remains validated over time. We often find that this area is overlooked during the initial validation of a system.
Section 5.2.2 Validation Procedures
1. We have not encountered the term “validation procedures” in this context and we feel that it requires further explanation. For the majority of the validation activities described in the Validation Plan, there needs to be a plan for each activity which describes the steps to be followed, this is executed, and the results are recorded in an activity summary report. The report is approved before the next activity described in the Validation Plan can be initiated. Examples include system testing, installation qualification, operational qualification, etc. We feel that section 5.2.2 should be expanded to convey this concept.
Section 5.2.3 Validation Report
1. As stated above, the majority of activities described in the Validation Plan require individual activity summary reports corresponding to that activity plan, for example, Installation Qualification Summary corresponding to the Installation Qualification Plan. In our opinion, it would be impossible to include all these “test results” in the final Validation Report. We agree that a Validation Report is required but that it should contain a summary of the validation activities described in the Validation Plan and how they were executed. It should a “provide a roadmap” to the other validation documentation.
2. We suggest adding that the Validation Report should describe the activities that will be performed to ensure that the system remains validated over time. We often find that this area is overlooked.
3. We would like to see a qualifier added to the sentence “Whenever possible, test results should be expressed in quantified terms rather than stated as pass/fail.” If the expected result is clearly stated in quantified terms, we believe that it is acceptable to just specify “pass/fail” indicating that the expected result was achieved. Requesting that the tester document test results that can be specified ahead of time is time consuming and in our opinion does not add value to the testing. Obviously where the expected results cannot be stated in quantified terms, we agree that actual results must be recorded.
Section 5.3 Equipment Installation
1. We support the statements in this section.
Section 5.4 Dynamic Testing
1. Add dynamic testing to the glossary.
2. We feel it would be beneficial to include an opening paragraph that describes the different levels of testing and provides some context to the following sections. This should define the type of testing, what is tested, when it is performed, who performs it, who reviews and approves it, and what documentation is produced, e.g., plan, results, summary including the resolution of any test failures.
Section 5.4.3 How test results should be expressed
1. As stated above, we would like to see a qualifier added to the sentence “Quantifiable test results should be recorded in quantified rather than qualified terms.” If the expected result is clearly stated in quantified terms, we believe that it is acceptable to just specify “pass/fail” indicating that the expected result was achieved. Requesting that the tester document test results that can be specified ahead of time is time consuming and in our opinion does not add value to the testing or the review and evaluation. Obviously where the expected results cannot be stated in quantified terms, we agree that actual results must be recorded.
Section 5.5 Static Verification Techniques
1. We support the statements in this section.
2. Add system level functional testing to the glossary.
Section 5.6 Extent of Validation
1. We strongly support the concept of risk analysis to determine the extent of validation required.
Section 5.7 Independence of Review
1. We totally agree and support the concept of independent review during the validation process. However, we feel the statement “…computer system validation should be performed by persons other than those responsible for building the system” is misleading. The validation process corresponds to the lifecycle of the system and includes many phases. Some of these phases are the responsibility of the persons who built the system, although they would obviously not be reviewing their own work. For example, code reviews should be performed by someone other than the person who wrote the code, test results should be reviewed by someone other than the person who performed the testing. We are concerned that the statement highlighted above will be misinterpreted by persons who develop software, including software vendors, into thinking that they have no responsibility for any aspects of computer system validation. We of course agree that the end users of the system are ultimately responsible for ensuring that the system is validated prior to using it for regulated activities.
Section 5.8 Change Control (Configuration Management)
1. Change control and configuration management need to be defined in the glossary. Although we agree that they are interconnected, we feel that they are two different concepts. Change control is the process of defining the change, determining the impact, specifying the change steps including testing, getting approval for the change, implementing the change, and documenting the result. Configuration management is ensuring that for any point in time, the configuration of the system is known and documented.
2. We agree with the statements in the first paragraph.
3. We feel that paragraph 2 is dangerous as it implies that contractors or vendors may be given remote access to validated computer systems to perform upgrades, fixes, etc. We feel that this should be avoided in all but emergency situations. Remote access must be controlled by persons responsible for the data and access only given to qualified individuals after a change has been appropriately reviewed and approved.
4. We agree with the statement in the third paragraph. We request that regression testing be added to the glossary.
Section 6 Special Considerations
1. We support the inclusion of this section regarding special considerations.
Section 6.1 Commercial, Off-The-Shelf Software
1. We feel the statement “…the end user’s validation approach for off-the-shelf software is somewhat different from what the developer does because the source code and development documentation are not usually available to the end user” is misleading. We would like the statement above to be changed to “…the validation approach for off-the-shelf software is somewhat different to custom developed software because the source code and development documentation are not usually available”. In our experience, it is not only end users that are involved in the implementation of COTS and the validation approach differences go beyond end user activities.
2. The statement “End users should also be able to validate off-the-shelf software by performing all of the following” is misleading. The areas listed are only a subset of the validation activities that need to be performed. Items that are missing include IQ/OQ of hardware and software, user training, SOPs, etc.
Section 6.1.1 End User Requirement Specifications
1. We agree with this section.
Section 6.1.2 Software Structural Integrity
1. We recommend changing the statement “..structural integrity by doing all of the following” to “..structural integrity. Activities include the following”. This allows alternative activities that produce the same outcome to be considered.
2. Some COTS vendors will not allow evaluation of their software development activities. If “all of the following” remains in this section, it would eliminate use of these systems even though, as stated in 6.1.3, it may be possible to do additional functional testing to compensate.
Section 6.1.3 Functional Testing of Software
1. We agree with the statements in this section.
Section 6.2 The Internet
1. We agree with the statements in this section.
Section 6.2.1 Internet Validation
1. “as computer system”, should read “as a computer system”.
2. We agree with the statements in this section.
Appendix A – References
We would prefer that the list of references be reduced to include only those that are current and those that support FDA’s current thinking on computer systems validation.
We hope these comments will be useful to the Agency in developing the final guidance.
Sincerely,
Kate Townsend
Vice President Regulatory Compliance and Validation
Taratec Development Corporation
Suite 111, Three Tower Bridge,
Two Ash Street, Conshohocken, PA 19428
Tel: 610-818-1372 Fax: 610-818-1201 email:ktownsend@taratec.com
|
|