M E M O R A N D U M

DEPARTMENT OF HEALTH AND HUMAN SERVICES PUBLIC HEALTH SERVICE FOOD AND DRUG ADMINISTRATION CENTER FOR DRUG EVALUATION AND RESEARCH

                                                                             

DATE:                        February 2, 2004

 

TO:                              Paul Seligman, M.D., M.P.H., Acting Director

                                    Anne Trontell, M.D., M.P.H., Deputy Director

Office of Drug Safety

Immediate Office, HFD-400

 

THROUGH:                Mark Avigan, M.D., C.M., Director

                                    Office of Drug Safety

                                    Division of Drug Risk Evaluation, HFD-430

 

FROM:                        Cynthia Kornegay, Ph.D., Epidemiologist

                                    Office of Drug Safety

                                    Division of Drug Risk Evaluation, HFD-430

                                   

SUBJECT:                  PID D030417

Drug: Isotretinoin

Topic: Review of Prescription Compliance Survey to Measure Compliance with Isotretinoin Qualification Stickers

 

 

EXECUTIVE SUMMARY

 

The primary purpose of the Prescription Compliance Survey (PCS) is to measure compliance with the isotretinoin qualification stickers.  In addition to compliance with the sticker program, the PCS also attempts to measure the completeness and accuracy of stickers from prescriptions filled at U.S. pharmacies.  The PCS is a retrospective, repeated measures survey that will ultimately include 6,000 randomly selected U.S. pharmacies.  Over a 2 year period, 750 pharmacies are recruited to participate each quarter. 

 

In conjunction with the survey, an audit is being conducted to validate the main survey results.  The design of the audit was for a random sample of 15% of PCS respondents to submit photocopies of the Accutane prescriptions used in the analysis.  In addition to the photocopy audit, the implementation plan for the PCS included a field audit of 20% of the audited pharmacies as a gold standard for data verification. 

 

The results show a very high rate of compliance with sticker use across all five survey waves, which consistently exceed the primary objective of 90% complete and correct prescriptions.  Results were consistent across gender, payer type, and age.  There were some differences in pharmacy strata specifically for prescription volume and population density. In the June 2002 survey, pharmacies with a volume of 2500 – 4999 prescriptions per month were more likely to receive Accutane prescriptions with incomplete stickers (missing gender and/or prescription date) than pharmacies with either a higher or lower prescription volume. For all of the surveys except June 2003, rural pharmacies were more likely than urban stores to receive an Accutane prescription without a sticker.  This difference was statistically significant for the first three survey waves.  The trend was apparent in the March 2003 results as well, but did not achieve statistical significance.  In the March and June 2003 surveys, rural pharmacies were more likely than urban stores to receive an Accutane prescription with an incomplete qualification sticker.  This pattern is not apparent prior to March 2003, but does achieve statistical significance in the June 2003 results. There do not appear to be any differences in female patients across the survey waves, nor were there trends by age or payment type.

 

Similar to the PCS, the audit shows a high rate of compliance and completeness among the validated prescriptions.  However, the implemented audit recruiting method appears not to be random, which is an important departure from the study design.  Since the sponsor does not describe the implemented recruiting method, the utility and/or applicability of these data are questionable.  If the audited pharmacies are not a random sample of the overall PCS sample, then the audit cannot contribute to validating the results found in the survey.

 

The two major limitations of the overall PCS are the low pharmacy response rate, and the low number of prescriptions captured for analysis.  Although more than 750 pharmacies were recruited for each wave of the audit survey, there have not been 750 responses to date.  In addition, during the third wave of the study, four pharmacy chains (Walgreens, CVS, Eckerd, and Rite Aid) and one retailer (Wal-Mart) asked to be removed from the study.  These stores represent some of the largest pharmacy chains and pharmacy retailers in the U.S, and their removal may have compromised the ability of the PCS to obtain the necessary number of prescriptions for a valid analysis.

 

Overall, these serious problems in the survey implementation and response rate make it unclear if the survey is truly representative of the national picture, or if it is even achieving the stated objective of measuring sticker compliance. 

 

The PCS is an indirect measure of physician compliance with S.M.A.R.T. program. The pharmacies are middlemen, and unless the corporate, chain, or insurance reimbursement policy dictates compliance with the S.M.A.R.T. program, pharmacies can dispense isotretinoin without the sticker.  In addition, the pharmacies can only influence physician compliance or participation by refusing to fill prescriptions without a sticker meeting SMART requirements.  Finally, given that this is an indirect measure of physician compliance with the pregnancy prevention measures in SMART etc, without directly asking doctors to confirm their level of participation with various sticker-indicated practices, a high compliance percentage can be a misleading indicator of physician compliance.

 

INTRODUCTION

 

The primary purpose of the Prescription Compliance Survey (PCS) is to measure compliance with the isotretinoin qualification stickers.  In addition to compliance with the sticker program, the PCS also attempts to measure the completeness and accuracy of stickers from prescriptions filled at U.S. pharmacies.  Overall compliance is defined as the proportion of prescriptions dispensed with a sticker versus the total number of dispensed isotretinoin prescriptions.  Completeness is defined as the proportion of correctly completed isotretinoin stickers versus the number of dispensed isotretinoin prescriptions with a sticker.  The sponsor document does not specify what a “correctly completed” sticker consists of, but, based on the information contained in the sticker, if the qualification date and gender are present, then it is considered correct and complete for females.  In the case of males, the gender and prescription date are required.  The term "qualification date" is the date that the patient successfully completed all pre-prescription activities, including signing the informed consent, receiving educational materials and contraceptive counseling, all pregnancy testing, and committing to use of two forms of effective contraception.  There is no explicit information concerning the pregnancy tests or their results on the sticker.

 

The purpose of the Prescription Compliance Survey Audit is to validate the main survey results.  The design of the audit was for a random sample of 15% of PCS respondents to submit photocopies of the Accutane prescriptions used in the analysis.  In addition to the photocopy audit, the initial implementation plan for the PCS included a field audit of 20% of the audited pharmacies as a gold standard for data verification. Due to legal limitations this part of the audit was not considered feasible and was therefore not implemented.^[1]

METHODS

 

Prescription Compliance Survey

 

The PCS is a retrospective, repeated measures survey that was designed to include 6,000 randomly selected U.S. pharmacies.  Over a 2 year period, 750 pharmacies are recruited to participate each quarter.  Pharmacies are independently selected from a national database of all pharmacies, and each pharmacy can only be selected once during the entire PCS survey period.  Stores are stratified by size, region, monthly prescription volume, and population density.  The database of potential pharmacies contains approximately 47,046 stores.  Five major pharmacy chains, representing 15,289 stores, were removed during the third wave of the survey.  Although this reduced the number of potential pharmacy participants to 31,757, an adequate number of stores remain available to complete the survey.  The sponsor does not discuss the impact of this withdrawal on the number of prescriptions available for analysis; however, this may be of greater concern than the unavailable pharmacies.  The PCS only covers retail dispensed prescriptions, which accounts for 92% of all isotretinoin prescriptions.  Of the remaining prescriptions, 4% are dispensed via mail order and 4% through other methods.^[2]  Each sample is created according to the strata percentages described in Table 1.  (The four geographic regions are described in Appendix 1.)

 

Prescription data covered four months during the first year of the SMART program: June 2002, September 2002, December 2002, and March 2003.  The first quarter of the second year of the program, June 2003, is also included in this analysis.  Table 2 shows the number of pharmacies recruited, the number responding, the number of Accutane prescriptions (Rx’s), and the number of stores reporting no Accutane prescriptions for each of the 1^st five waves of the PCS.

 

Table 2.  Cohort of PCS Participants
PCS Wave	Pharmacies Recruited	No. of Stores Responded	Total number of Accutane Prescriptions*	Stores with No Accutane Prescriptions
June 2002	869	221	319	111
September 2002	761	392	308	263
December 2002	759	444	331	329
March 2003	755	445	201	366
June 2003	868	469	181	390
*The number of reported or estimated Accutane prescriptions during the specified time period.  This is the total number of prescriptions that were available for analysis in that survey wave.

 

The sponsor reports that response rates were 25.4%, 52%, 58%, 59%, and 54% for each of the five waves.  Based on the low response rate for June 2002, the recruiting strategy was changed for subsequent surveys.  As a result, the participation rate doubled in September 2002, and has remained in the 50% range since that time.  The number of Accutane prescriptions captured, however, is declining, and the number of responding pharmacists that had no Accutane prescriptions in each wave – 53%, 67%, 74%, 82%, and 83% – is increasing.

 

In the third wave of the study, four pharmacy chains (Walgreens, CVS, Eckerd, and Rite Aid) and one retailer (Wal-Mart) asked to be removed from the study.  These stores represent some of the largest pharmacy chains and pharmacy retailers in the U.S.  Although, as the sponsor states, the number of stores that remained in the sample was more than adequate to obtain the required number of stores, the effect of the removal of these chains on the number of prescriptions obtained is potentially more severe.  To illustrate this, Table 3 shows the percentage of prescriptions accounted for by those five chains for December 2002.  (See Appendix 2 for a description of the AdvancePCS data resource.).  In the AdvancePCS data resource, these five chains account for approximately 40 to 45% of the total prescription volume for the survey months in question. 

 

 

The effect of the loss of these stores can also be seen in Table 4, which shows the mean number of prescriptions both overall and by store size (Independent/Small Chain, Large Chain) for each of the survey periods.  The analysis plan for the survey estimated that there would be a minimum of 2.55 prescriptions per pharmacy for each survey wave.  That level of prescription activity has not yet been seen in this survey.  The only rate that approaches the projected level is 2.40 prescriptions per store which was seen in large chain pharmacies during the first wave of the survey.  However, even that level was not sustained, since the next two waves show that the mean number of prescriptions for large chains was less than half of the rate seen in the first survey wave.

 

 

PCS Audit

 

A sample of PCS respondents agreed to submit photocopies of any Accutane prescriptions dispensed during the time period of interest.  The sponsor’s goal was to audit a random 15% of both prescriptions and pharmacies.  The actual recruiting method for the audit is not described in the sponsor’s report; however, it appears that the selection of audit participants is not random.  Evidence of this is provided by the increased participation in the audit to a high of 28% during the March 2003 survey wave.

 

The prescription photocopies were compared to the corresponding survey answers to determine the completeness and accuracy of the survey response.  The initial audit was designed to last through the first two waves of the survey, with adjustments to the survey methodology to be implemented after that period.  Table 5 indicates the number of respondents to the audit survey for the first 4 waves.

 

Table 5. Responses to Pharmacy Audit Survey
PCS Wave	PCS Responding Pharmacies	Pharmacies Recruited for Audit	Pharmacies responding to Audit	Accutane Prescriptions Audited	Pharmacies with no Accutane Prescriptions
June 2002	221	62	39	62	0
September 2002	392	75	69	44	54
December 2002	444	135	123	63	108
March 2003	445	141	128	36	105
June 2003	469	106	96	32	78

 

Table 6 shows the mean number of prescriptions of audited pharmacies overall and by store size. Similar to the overall survey, the highest mean number of prescriptions was captured in the June 2002 survey.  Of note is the extremely low number of prescriptions by independent pharmacies in the September 2002 survey, but this appears to be an anomaly.

 

 

RESULTS

 

Prescription Compliance Survey

 

Tables 7 through 12 highlight the primary results of the PCS survey.  For all of the survey waves, the number of prescriptions with a sticker was better than 95%, and the number of correctly completed^[5] stickers was above 90%, the sponsor’s stated goal.  The gender split of surveyed prescriptions was approximately 50% female for all waves, with between 2% and 6% of prescriptions not identifying the gender.  The average age of individuals whose prescriptions were surveyed was approximately 22 years old, with a median age of 17 – 18 years, and a range generally from 11 to 75 years.  (In the June 2002 survey, there was one reported age of 1 year and one of 3 years old.) 

 

 

 

 

 

Overall, the results were consistent across gender, payer type, and age.  There were some differences in the pharmacy strata, specifically for prescription volume and population density.  In the June 2002 survey, pharmacies with a volume of 2500 – 4999 prescriptions per month were more likely to receive Accutane prescriptions with incomplete stickers (missing gender and/or prescription date) than pharmacies with either a higher or lower prescription volume (89% complete for 2500 – 4999 prescriptions vs. 100% and 97% complete for 1 – 2499 prescriptions and 5000+ prescriptions, respectively).

 

Table 12 shows the percent compliance and completeness across all survey waves for rural vs. urban pharmacies.  For all of the surveys except June 2003, rural pharmacies were more likely than urban stores to receive an Accutane prescription without a sticker.  This difference was statistically significant for the first three survey waves.  The trend was apparent in the March 2003 results as well, but did not achieve statistical significance.  In the March and June 2003 surveys, rural pharmacies were more likely than urban stores to receive an Accutane prescription with an incomplete qualification sticker.  This pattern is not apparent prior to March 2003, but does achieve statistical significance in the June 2003 results.

 

 

PCS Audit

 

Table 13 shows the overall compliance (presence of a sticker) and completeness as reported in the PCS periodic summaries, and as calculated by applying the error rates observed in the audit to the main PCS sample. 

 

Applying the audit error rates to the reported compliance and completeness rates (instead of using the counts of non-complete or noncompliant audited prescriptions) provides an alternative estimate of the number of prescriptions that were compliant with the S.M.A.R.T. survey guidelines.  It is important to note that not all of the “errors” found in the audit are represented here. If a date was incorrectly reported, but the interval between the qualification date and the dispensing date was < 7 days, it was not considered as being in error.

 

 

The numbers and percentages of prescriptions that differed between what was reported and what was audited were also examined within each of the pharmacy selection strata.  For geography, population density, and monthly prescription volume, there were no striking differences between strata; however, the September 2002 survey had generally lower rates of agreement than any of the other surveys.  When examined by independent/small chain stores vs. large chains, along with a lower agreement rate in September 2002, there was a tendency for the independent/small chain stores to have lower agreement rates than the large chain stores.

 

DISCUSSION

 

Prescription Compliance Survey

 

The results show a very high rate of compliance across all five survey waves, which consistently exceed the primary objective of 90% complete and correct prescriptions.  There do not appear to be any differences in the percentage of female patients across the survey waves, nor were there trends by age or payment type.

 

At the pharmacy level, there was a trend towards lower compliance and completeness in rural vs. urban pharmacies, although the neither of those two metrics ever fell below 85%.  Since this is a voluntary system, and pharmacists can call and verify any missing information, it is also not clear that the dispensing the prescription with incomplete information was inappropriate.

 

The random sample design and the stratified recruiting strategy are strengths of this study.  If there had been a sufficient number of prescriptions captured, this study would have given a national picture of compliance with the Accutane qualification stickers.  However, there appear to be serious problems in the implementation of the survey.  The sponsor acknowledged that the first wave of the survey had some operational issues, resulting in a very low response rate for the June 2002 wave.  Several changes were implemented starting with the December 2002 wave, resulting in a higher response rate for the subsequent survey waves.  However, the response rates for the pharmacies and the number of Accutane prescriptions captured continue to fall short of the minimum projected amount needed for analysis based on the PCS analysis plan.

 

The effect of CVS, Eckerd, Rite-Aid, Walgreen’s, and Walmart dropping out in the third wave  (Table 3) of the study can be seen most easily by examining the mean number of prescriptions captured for each of the waves (Table 4).  It is interesting to note that although there is a drop as expected in the December 2002 survey, the mean number of prescriptions for the two prior waves were still lower than the targeted 2.55 prescriptions per store (1.90 and 0.79, respectively).  The result of these low response and prescriptions rates is that the study is underpowered, which makes drawing conclusions difficult, and makes national generalizations based on these conclusions (the overall goal of the PCS) unadvisable.

 

Prescription Compliance Survey Audit

 

When the audited pharmacies are examined according to the sample strata, as in the overall survey, high volume (5000+ prescriptions per month) and urban pharmacies tend to be underrepresented.  The representations of independent/small chain vs. large chains stores, and geographic area, are approximately the same as the sampling strata percentages.  Table 5 shows roughly the same pattern as for the overall survey (Table 2).  The decline is mostly seen until June 2003, where there appears to be an increase, but when the actual numbers are examined, this could be seen as capturing a larger percentage of a decreasing number of prescriptions.

 

Similar to the PCS, the audit shows a high rate of compliance and completeness among the validated prescriptions.  Even when the overall rates are adjusted to reflect the error rates seen in the audited prescriptions; both compliance and completeness remain above 90% for all of the survey waves.  However, there appear to be two problems with the audit as it was implemented.

 

In the first wave, only pharmacies with Accutane prescriptions responded to the audit.  The precise recruiting strategy is not described, but it is possible that stores were recruited based on the number of Accutane prescriptions they reported or estimated, and not randomly (as described in the analytical plan).  This situation did not occur in subsequent waves, so it appears to have been resolved.

 

The second issue is the participation in the audit rate for survey waves 2 through 5.  The design of the audit called for a 15% sample of responding pharmacies to be audited.  The number of pharmacies participating in the audit ranged from a low of 18% (June 2002) to a high of 28% (March 2003).  The number of prescriptions audited was between 14% (September 2002) and 19% (December 2002).  The recruiting strategy does not appear to have been altered even though the audit response was consistently high.  Again, the recruiting method is not specified, but it does appear not to be random.  Given the potential problems with the selection of the audited pharmacies, the utility and/or applicability of these data are questionable.  If the audited pharmacies are not a random sample of the overall PCS survey sample, then the audit cannot contribute to validating the results found in the survey.

 

Even if the audited pharmacies are a random sample of the survey, the results may underrepresent adherence to the SMART requirements. In the case of stickers that were not attached or not correctly filled out, the pharmacist could contact the doctor directly, confirm the prescription, and dispense the drug according to guidelines.  However, this might not be reflected in the survey or validation audit, since only the actual sticker is considered.  If this is the case, then pharmacy compliance may be at or above the calculated level.

 

CONCLUSIONS

 

Strengths

 

The PCS was designed to be a stratified random sample of U.S. retail pharmacies.  Once it was refined, the recruiting strategy appears able to consistently obtain close to the number of responded needed to have adequate power for analysis.  As designed and described in the analysis plan, the validation audits would have provided an excellent quality check on the main survey if done as planned.

 

Limitations

 

Two major limitations of the overall PCS are the low pharmacy response rate, and the low number of prescriptions capture for analysis.  Although more than 750 pharmacies were recruited for each wave of the audit survey, there have not been 750 responses to date.  The most successful wave yielded a response rate of 60%, which is similar to results obtained from a survey of pharmacists^[7], but generally lower than other surveys that combine mail and telephone methods^{[8], [9].}  The sponsor estimated that the response rates would range from 60% to 75%, and include between 450 and 525 pharmacies, and 1,350–1,575 Accutane prescriptions.  While the last two survey waves approach the lower bound of the baseline response rate target.  However, the number of Accutane prescriptions submitted is only 15%–20% of the projected Accutane prescription rate.

 

When the response rate is examined by the predetermined strata, a consistent picture emerges.  Large, urban stores are consistently under-represented, as are high-volume pharmacies.  The geographic representation of the responding population is the most similar to that of the recruited sample, although the West tends to be slightly under-represented (2 – 4%).

The same picture is not apparent when the mean number of prescriptions is examined across all of the pharmacy strata.  Despite the response rate problems, the June 2002 survey had the highest level of mean prescriptions per pharmacy (1.9) both overall and for each of the pharmacy strata.  When combined with the low response rate for this wave (25.4%), the possibility that pharmacies with higher numbers of Accutane prescriptions were more likely to respond cannot be ignored.  Even with the trend towards less prescriptions for each pharmacy across all of the other waves, the mean number of prescriptions both overall and within each of the pharmacy strata are generally more similar to each other than to the results from the first PCS survey.

 

In the third wave of the study, four pharmacy chains (Walgreens, CVS, Eckerd, and Rite Aid) and one retailer (Wal-Mart) were removed from the list of pharmacies that could be recruited.  These stores represent some of the largest pharmacy chains and pharmacy retailers in the U.S., and Table 3 illustrated the number of Accutane prescriptions handled by these five chains in AdvancePCS, a large pharmacy benefit management organization.  Although this data resource is not nationally representative, it highlights the possibility that the exclusion of these chains eliminated a significant portion of the prescriptions available for analysis.

 

A related problem is the declining number of prescriptions captured in the PCS survey (Table 4).  Although the report makes note of the excluded pharmacies in the third phase of the survey, the decline had begun prior to the five pharmacy chains’ decision to not continue participation.  Based on the number of prescriptions gathered in each survey wave, even if all of the recruited pharmacies responded, the number of prescriptions would still be insufficient for analysis.  For example, using the results of the June 2003 survey, approximately 4,150 stores (an increase of 550%) would have been needed to achieve an adequate sample size, assuming a 60% response rate.  This is possible given the number of stores remaining in the sample, but it would make the logistics of the survey more complicated.

 

From the survey instruments included in the report, and an examination of the sponsor’s description of the PCS, it is not clear if there are a large number of prescriptions being eliminated due to the pharmacy selection, data collection, data cleaning, or other factors.  While whatever problem that caused this initial decline was overshadowed and exacerbated by elimination of the five chains in the December 2002 survey, it is doubtful that the basic problem has been addressed.

 

Overall the lower than expected response rates for the entire survey for both prescriptions and pharmacies indicate that there may be a fundamental problem in the sample size and power calculations, and in the way the survey is currently being implemented.  The pilot study resulted in a very high response rate, which does not seem to have been confirmed once the survey was implemented.  There are several non-statistical factors which might contribute to this result, such as the method and scope of recruiting in the pilot versus the actual survey, the number of market survey requests received by the pharmacies, and changes in corporate policy regarding participation in market surveys.

 

With regards to the PCS audit, there appears to be a fundamental problem with the recruiting strategy that was implemented.  Participant selection in the audit surveys has not been according to the plan set forth by the sponsor.  In the June 2002 wave, it appears that the audit pharmacy selection was somehow biased towards stores with higher numbers of Accutane prescriptions.  In the subsequent waves, the method of selection is not specified, however, since the participation rate is consistently higher than the targeted rate (up to 28%).

 

Recommendations for the PCS, and audit, and also suggestions for making the results of future survey waves easier to analyze and interpret are included in appendix 3^[10]. 

 

OVERALL CONCLUSIONS

 

The overall purpose is to measure physician and pharmacy compliance with the qualification stickers, and, based on the survey results, both are quite high and meet or exceed the sponsor’s stated goals.  The secondary goals of accuracy and completeness are also achieved.  However, serious problems in the survey implementation and response rate make it unclear if the survey is truly representative of the national picture, or if it is even achieving the stated objective of measuring sticker compliance.  In addition, the implementation of the data validation audit appears to differ significantly from the analysis plan, making its interpretation and usefulness questionable.

 

It is important to remember that the PCS is an indirect measure of physician compliance with S.M.A.R.T. program. The pharmacies are middlemen, and unless the corporate, chain, or insurance reimbursement policy dictates compliance with the S.M.A.R.T. program, pharmacies can dispense isotretinoin without the sticker.  Pharmacies can only influence physician compliance or participation by refusing to fill prescriptions without a sticker meeting SMART requirements.  Finally, given that this is an indirect measure of physician compliance, without directly asking doctors to confirm their level of participation, a high compliance percentage can be a misleading indicator of physician compliance.