MQSA Facility Satisfaction Survey - First Segment
In the Spring of 2001, the Division of Mammography Quality and Radiation Programs (DMQRP), Center for Devices and Radiologic Health (CDRH), Food and Drug Administration (FDA) received approval from the Office of Management and Budget (OMB) to conduct a customer satisfaction survey under OMB No. 0910-0360. Approximately 10% of the Mammography facilities that are accredited and certified by FDA under the Mammography Quality Standards Act of 1992 (MQSA), as amended by the Mammography Quality Standards Reauthorization Act of 1998 (MQSRA), were surveyed.
The survey served as a follow-up to a similar survey conducted in 1997 of facilities regulated under the MQSA Interim Regulations. Therefore, most of the questions included in the 2001 survey were also asked in the 1997 survey. This duplication allows for direct comparison and monitoring of changes in the inspection program resulting from the implementation of the MQSA Final Regulations on April 28, 1999, and 1997 survey findings. In addition, questions reflecting changes in the way FDA communicates with mammography facilities were included.
The purpose of the survey was to collect information to help FDA evaluate and improve, where needed, both the inspection process and communications with MQSA-regulated facilities. The comparison of the 2001 survey results to those of the 1997 survey is expected to help determine where progress had been made, and to solicit comments and recommendations from facilities on the MQSA inspection process.
All surveyed facilities had been inspected at least once under the Final Regulations and most of them had also been inspected under the Interim Regulations prior to completing the survey. One thousand facilities were randomly selected from the approximately 9,600 MQSA-certified facilities.
To ensure confidentiality of facilities’ responses, FDA/DMQRP used SciComm, Inc., of Bethesda, Maryland, an independent contractor, to conduct and report on the findings of the survey.
In May 2001, SciComm, Inc. of Bethesda, Maryland, on behalf of DMQRP initiated the Facility Satisfaction Survey of 1,000 mammography facilities. The goal was to achieve a 65% response rate. The initial mailing of the survey instrument to the selected facilities was followed up by a second mailing to those facilities that had not responded to the first mailing.
By the final closing date, 74% of the surveyed facilities had responded to the survey. After the closing date, an additional 2% responded. While these survey responses could not be included in the data analysis, they were examined and found to be consistent with the overall findings. The open-end comments from the late responses have been included in the final report and are identified as being from surveys outside those analyzed.
The survey questionnaire (Appendix A) covered five general areas: 1) MQSA Information Resources, 2) Pre-inspection Process, 3) Inspection Process, 4) Professionalism of the Inspector, and 5) Inspection Results as Provided in the Exit Interview. The section on Respondents’ Demographics describes the survey sample.
In July 2001, FDA received a report of the initial findings (Appendix B). An additional analysis of this data was conducted using cross-tabulation to examine correlations between various findings, i.e., size of facility and access to the Internet. This report presents those findings.
Throughout this report, percentage response numbers will be referenced. These percentage numbers represent the breakout (or cross-tabulation) of responses received for both the primary and secondary topics under discussion. These breakouts have been tested for statistical significance.
For example, in looking at the information about visiting the Mammography web site from work, we see that there is a statistically valid indication that those from larger facilities were more likely to have visited the web site from work.
The table below shows the breakout of the secondary topic (number of units) against the primary topic (have visited web site). The numbers that are statistically significant have been marked with a star (*). The number of respondents is shown in parentheses.
|Have Visited the
Mammography Web Site
|Number of Mammography Units at Site
|Yes: 38.9% (284)||36.1% (176)||38.8% (54)||53.5%(*) (54)|
|No: 61.1% (447)||63.9%(*) (311)||61.2%(*) (85)||46.5% (47)|
Note that while 731 respondents answered the question about visiting the web site from work, only 727 also indicated how many mammography units their facility has.
- 284 of 731 respondents said that they had visited the web site from work (38.9%);
- 487 of 727 respondents said their facilities had 1 unit (67.0%), 139 had 2 units (19.1%), and 101 had 3+ units (13.9%);
- A higher percentage of those from facilities with 3+ units had visited the web site than the percentage of those from facilities with 1 or 2 units.
Therefore, in this case we would say:
- Respondents from facilities with 3+ mammography units were more likely to have visited the web site from work (53.5%) than those from facilities with 1 unit (36.1%) or 2 units (38.8%).
MQSA Information Resources
Internet Access and Mammography Web Site
Given that FDA is now publishing all the MQSA information exclusively in an electronic format on its Mammography web site, it is noteworthy that just 52% of the respondents were aware that this information is published only on the web site.
It is significant that the Radiologic Technologists (RTs), who account for 81% of the respondents and generally represented the facility at the inspection, were less likely to be either aware that information is published only on the Mammography web site or to have access to the Internet either at work or home/elsewhere than the Administrators, who account for 10% of the respondents (see page 10).
Only 60% (436) of the respondents reported that they had access to the Internet at work; of those, only 63% (37% of all respondents), had visited the web site from work and 34% (20% of all respondents) had visited it from home/elsewhere. Of the 80% (591) who reported that they had access to the Internet from home or elsewhere, only 44% (35% of all respondents) had visited the web site from home/elsewhere and 41% (33% of all respondents) had visited it from work. In fact, less than 40% of all respondents had actually visited the website.
This situation has improved since 1997 when 63% of the respondents stated that they had no access to the Mammography web site — that number is now down to 24%. However, in 1997, FDA was printing and mailing information to the facilities. By 2001, the only place for facility staff to access MQSA information was on the web site, which means that 24% of the respondents did not have access to new and updated MQSA information.
Generally speaking, those who had visited the web site from work tended to be from larger facilities, knew that the information is published only on the web site, and spent more time preparing for the inspection. Those who had visited the web site from home/elsewhere also knew that information was only published on the web site, but were less likely to have access at work.
Respondents were asked to rate four publications in both the 1997 and 2001 surveys. In each case, the percentage of “No access” responses in 2001 was 2–3 times greater in 2001 than in 1997. The distribution pattern across the sample indicates that these respondents tended to be from smaller facilities.
As in 1997, the most highly rated publications were “Preparing for MQSA Inspections” and Mammography Matters. It was apparent from the responses that many of the respondents were not aware that publication of the Mammography Matters newsletter has been suspended. For the most part, respondents who rated these publications highly were evenly distributed across the sample. However, as would be expected, those who had visited the web site from both work and home/elsewhere and knew that information is only available electronically rated them higher than those who had not visited the web site.
Respondents rated “The Mammography Facility Survey, Equipment Evaluation, and Medical Physicist Requirements Under MQSA” more useful in 2001 than in 1997.
The Federal Register continued to be rated the least useful of the publications, although those from larger facilities tended rate it higher.
Between those who reported that they had no access to the Mammography web site (24%) and those who didn’t use it (24%), nearly half of the respondents had not visited the web site. They tended to be from smaller facilities and did not know that MQSA information is published only on the web site.
The MQSA Hotline was rated higher in the 2001 survey than in the 1997 survey, although 40% of the respondents in both surveys had not used it. Those who used the Hotline rated its responsiveness higher in 2001 than the 1997 respondents did.
Information provided by the inspector was also rated higher in 2001 than in 1997. In part, this increase may be based on the fact that the inspector is the primary source of information for those without access to the web site. It is also indicative of a good inspector/facility relationship.
The 26% of respondents who said they didn’t have access to the Policy Guidance Help System is in line with those who do not have access to the Internet. An additional 36% said that they had access but didn’t use it. However, of the 22% of the respondents who said they had used it, an impressive 93% found it useful.
E-mail notification of MQSA-related information is another resource dependent upon Internet access. However, more respondents said they didn’t have access to e-mail notification (43%) than those who said they did have access to the Internet at work (41%) or at home/elsewhere (20%), indicating that access to the Internet does not translate into e-mail accounts. Those with no access tended to be from smaller facilities.
Notice of Inspection and Rescheduling of Patients
In 1997, respondents indicated that they needed more notice of the inspection. It is obvious from the 2001 survey that FDA responded to this request. Those facilities that received 14 or more days notice increased to 55% from 39%. Concurrently, those facilities that received less than 10 days notice dropped from 31% to 17%. Most (95%) facilities indicated that they had received enough notice. Those who felt they did not receive adequate notice received 1–9 days notice.
More than half (52%) of the facilities had to reschedule patients. Facilities with 1–2 units were more likely to have rescheduled patients than larger ones and were also more likely to say that they had not received enough notice.
Based on respondents’ comments, the need to reschedule patients is the primary reason given for asking for more advance notice. They also indicated that it was very helpful when the inspectors were able to accommodate their scheduled patient load and/or come in on a day when no patients were scheduled. However, others commented that their inspectors were not flexible in scheduling the appointment even when the facility representative was scheduled to be on vacation.
Preparing for the Inspection
The 2001 survey respondents reported that they spent 1.6 more hours preparing for the inspection than did the 1997 respondents. This increase in preparation time can be attributed to the fact that the average number of mammography units and mammograms performed both daily and on inspection day have increased. In addition, inspections are now being conducted under the Final Regulations.
When characterizing the time spent preparing for the inspection, there was a dramatic decline in the “Minimal” rating (from 64% to 21%), while those who were “Neutral” about the amount of time required increased from 35% to 69%. However, there was a significant increase in those who rated the time “Excessive” (from 1% to 10%), especially by those who had spent 10 or more hours preparing.
Equipment and Activity
There has been a 21% increase in the average number of mammography units at the surveyed facilities since 1997 (from 1.4 units to 1.7). Concurrently there has been a 28% increase in the average number of mammograms performed on a daily basis (from 16.2 to 20.7), and an even greater increase (38%) in the average number of mammograms performed on inspection day (from 8.7 to 12.0).
With all of this increased activity, there has been a minimal (5% increase in the average number of hours to complete the inspection (from 5.7 hours to 6.0). And, in spite of the characterization of time spent preparing for the inspections (those who rated it “Excessive” jumped from 1.1% in 1997 to 10.8% in 2001), the average time spent preparing for the inspection increased by only 18% (from 8.7 hours to 10.3), less than that for either the number of units or the number of performed mammograms.
In both surveys, 92% of the respondents were told how long their inspection should take and 91% reported that the inspection was completed within the expected time.
In the 2001 survey, respondents were asked about the amount of time it took to complete the inspection of each mammography unit — an average of 2.1 hours. Units at larger facilities were more likely to be inspected in less than 1.5 hours, while it tended to take more time at smaller facilities. Respondents indicated that they believed the time spent inspecting their equipment to be either “Minimal” or “Neutral.” Several respondents suggested that the inspectors use the medical physicist’s report instead of duplicating many of the same tests.
Recent and Past Inspections
There was a 10 percent shift between the 1997 and 2001 surveys in the rating of the inspection process for the most recent inspection from “Excellent” to “Fair.” However, the inspection process was still highly rated with 65% of the respondents rating the process “Excellent” and 34% rating it “Fair” in the 2001 survey.
When asked to compare their recent inspection to the previous inspection (the 1997 survey asked for a comparison to their first inspection —so there is not a direct comparison), there was a shift from “Better” to “Same.” In 2001, 67% of the respondents said it was the “Same” compared to 49% in 1997. This could indicate that the inspection process is stable and facilities know what to expect.
There were a number of comments received about the inspection process. Two issues emerged as recurring themes:
- Inspection should take less time and be more efficient;
- Inspectors should be consistent and less strict and critical.
Respondents again asked for checklists and standard formats for organizing materials for the inspectors. They also suggested that personnel records, especially the initial qualifications, be housed in a database so that only the CEUs, license, etc. would have to be reviewed. One respondent suggested that when inspectors are inspecting multiple sites with the same personnel, they should not have to re-examine the same personnel credentials at every site.
Professionalism of the Inspector
There was essentially no change in how the respondents rated the professionalism of the inspectors between 1997 and 2001. In both surveys, inspectors received high ratings across all attributes.
Suggestions and comments offered in 1997 reappeared in 2001 and addressed such issues as: inspectors being knowledgeable about mammography, consistency among inspectors’ expectations and interpretations, and standardized forms or formats for facilities to use.
As in 1997, 85% of the respondents were comfortable with having the same inspector again. In fact, many respondents felt that it helped streamline the inspection process because of the inspector’s familiarity with the facility and the staff’s familiarity with the inspector.
In 1997, respondents were given the option of selecting “Not sure” when asked if the MQSA Hotline was a good way to provide inspector feedback — 63% selected that option. In 2001, they were forced to select between “Yes” and “No.” Although 90% of the respondents said that the MQSA Hotline was a good way to provide feedback about their inspector, the comments do not fully support this response. A number of respondents were not aware that the Hotline could be used for this purpose; others were concerned about confidentiality and fear of reprisals. In addition, 40% of the respondents had not used the Hotline at all.
When asked for suggestions about other ways to provide inspector feedback, respondents offered: comment cards to be completed and mailed in, surveys to be completed and returned, and evaluation forms.
Inspection Results as Provided in the Exit Interview
Exit Interview and Citations
Inspectors conducted exit interviews at 93% of the facilities upon completion of the inspection and respondents indicated that they were useful.
Only 38% of the facilities received citations. Of those cited, 97% understood the reasons why and 95% knew how to respond to them.
The majority of the respondents (92%) said that they received an initial report at the conclusion of the inspection and 36% said that they received their final report immediately, down from 61% in 1997. There was an increase from 30% to 50% in those respondents who said that they received their report within one month.
In both surveys, nearly three-fourths of the respondents said that the inspection results were “Very” useful in identifying areas for improvement and 80% believed that the results were “Very” valid. In 2001, respondents were asked if they believed that the records were correctly evaluated and 82% said that they were.
There was a significant shift in how the respondents rated the usefulness of the cover letter that explains how to respond to inspection findings. Only 67% found it “Very” useful in 2001, down from 88% in 1997.
When asked if the inspector-provided information would be helpful in future inspections, there was a 10 percent shift from “Very” to “Somewhat” helpful. There were similar rating shifts in questions regarding the educational and beneficial aspects of the inspection. This result is not unexpected as the inspection process has now been in place for a number of years and most respondents are familiar with the process. The fact that only 38% of the facilities received citations confirms that most facilities understand what is expected and prepare appropriately.
As a result of problems that have surfaced when facilities close without fulfilling their MQSA responsibilities, FDA wanted to know if facilities would find it helpful for inspectors to review the steps they should take if their facility decided to stop performing mammography or close. While some respondents indicated that they didn’t understand why they were being asked that question, 69% said “Yes.”
When respondents were asked what other information they would like to learn from the inspection that is currently not provided, information on updates and guideline changes received the most comments. In addition to wanting updated information, respondents also wanted to know where to get forms and samples of acceptable policy and procedures statements.
A number of respondents stated that they were comfortable with the inspection process and very complimentary about their inspectors.
In both 1997 and 2001 surveys, four-fifths of the respondents were Radiologic Technologists.
More than 95% of the respondents had been the facility representative for the latest inspection surveys.
Eleven percent of the respondents said that their facility had responded to the 1997 survey, while 72% stated that they were not sure.
The geographic distribution of the responding facilities was in accordance with the U.S. Census 2000 population distribution.
The majority of respondents reported that their facilities had one mammography unit.
The average number of mammograms performed on a daily basis was 20.7, while the median number was 14. The distribution of mammograms performed on a daily basis was fairly evenly spread among facilities performing 0–9 mammograms (29%), those performing 10–19 mammograms (35%), and those performing 20 or more mammograms (36%).
The average number of mammograms performed on inspection day was 12, while the median number was 6. The distribution of mammograms performed on a daily basis was fairly evenly spread among facilities performing no mammograms (29%), those performing 1–10 mammograms (39%), and those performing 11 or more mammograms (33%).
This distribution closely parallels the number of mammograms performed on a daily basis.