• Decrease font size
  • Return font size to normal
  • Increase font size
U.S. Department of Health and Human Services

Food

  • Print
  • Share
  • E-mail

Public Workshop on Measuring Progress in Food Safety - Current Status and Future Directions (Transcript)

<< Meeting Main Page

March 30, 2010

Public Workshop on Measuring Progress in Food Safety: Current Status and Future Direction

This page is a transcript of the printable PDF.


 

DR. BERU: Good morning. If everyone will take their seats, we'll get started soon. Thank you. Good morning again. My name is Nega Beru. I'm the Director, Office of Food Safety, at FDA's Center for Food Safety and Applied Nutrition. On behalf of Dr. Stephen Sundlof, who is the Director of the Center, I'd like to welcome you to this very important workshop on Measuring Progress in Food Safety: Current Status and Future Directions. Before we get started, our sign language interpreters are Heidi Johnson and Megan Dabbs. Is there anyone -- does anyone need the support of an interpreter? I don't see anyone, but they'll remain available for a while. Okay?

There will be a one-hour question-and-answer session at the very end of the workshop. Most sections of the workshop also will be followed by brief questions and answers, as time permits, and as announced by the moderators. The meeting is being transcribed,so please clearly state your name and organization before you speak to assure that the information is captured for the transcript.

If you need help, look for the FDA staff. We're all wearing yellow badges. And you all have received a packet of information. The packet contains the agenda, the background information, the Federal Register notice announcing this workshop, the attendee list, and a list of nearby restaurants.

With that, I would now like to introduce Mike Taylor. Mike Taylor, an attorney who was named Deputy Commissioner for Foods at the U.S. Food and Drug Administration in January 2010, is the first to hold that position, which was created along with the new Office of Foods, in August 2009, to elevate the leadership and management of FDA's Foods Program.

He is a nationally recognized food safety expert, having served in high-level positions both at FDA and USDA, as a research professor in academia, and on several National Academy of Sciences expert committees. As Deputy Commissioner for Foods, Mr. Taylor provides leadership and direction to all of FDA's Foods Program. Mike?

MR. TAYLOR: Thank you, Nega, and good morning, everybody. I'm at the podium now in the role of an introducer. You know, when you introduce somebody, you always say what a pleasure it is to introduce them, and I just have to say it really is a pleasure for me to be able to introduce to you this morning Tino Cuéllar, who is here from the White House this morning to provide opening remarks.

Tino's title is Special Assistant to the President for Justice and Regulatory Affairs. But more important even than that, for our purpose this morning, is that he is the President's point person on food safety. In this capacity, Tino works very closely with the President's food safety working group, with the job of really ensuring this is the role of the working group -- Tino helps ensure that we stay focused on ensuring that food safety gets the priority it deserves in the Administration, at the White House, and that it deserves, really, as one of the President's top priorities.

Tino would be the first, I think, to acknowledge that he's not a lifelong food safety professional like so many of the people in this room. But he does come to his role with really great experience and expertise in making government work to solve complex problems, of which, I think we'll acknowledge, food safety is one.

Tino has his PhD in political science from Stanford and his law degree from Yale, and is a professor, actually on leave currently as a professor - from his role as a professor at the Stanford Law School, where his research and his teaching focused on how government institutions can do their jobs best, including tackling very complex regulatory problems. He's also worked for several years as senior advisor to the Treasury Department's Under Secretary for Enforcement.

So Tino Cuéllar brings great training, great expertise, and experience to his job at the White House. But even more importantly, and all of us who have worked with him see this, Tino brings passion to his job, to his work, and particularly to food safety. Tino really does understand the importance of food safety to the country, certainly in a public-health and a global economic sense, but also to families who are affected by foodborne illness.

So we really are fortunate. I'm very grateful that Tino is investing so much time in achieving the President's food safety goals, working with the Food Safety Working Group, with the agencies, and with the Congress to achieve our goals. I think it's fair to say, and it’s certainly the way we look at it at FDA, that in Tino we have a friend at the White House, and that can be a very valuable thing to have in today's world.

So Tino, I just can't thank you enough for being here, and look forward to your remarks.(Applause)

DR. CUÉLLAR: Good morning. Welcome. I want to thank Mike for that very kind introduction and acknowledge all the help that Mike has given us in his very important role at the FDA, and Jerry Mande of USDA, Peggy Hamburg, Kathleen Sebelius, Tom Vilsack -- these are some of the folks who really are making the Administration's public-health and food safety agenda a reality, and working very hard in bringing their experience and their judgment to this.

I have to admit that when I first walked in, I thought I was in the wrong meeting. I looked around and I saw just way too many people for what I thought was going to be a discussion about metrics, which, to many people around the country, would be the absolutely driest, most boring thing you could possibly imagine. So I first of all congratulate you for putting your priorities where they ought to be and for being here.

The reality is that every one of us is touched by this subject. And when I've spoken to families, when I've spoken to scientists about this topic, I often explain to them that my kids, one of them six and one of them three, don't really understand anything about my job, except this: they understand that the food that comes to their table -- they're beginning to understand, at least my six-year-old daughter -- that food does not magically appear there. It doesn't just simply emerge from the table. That many people are involved in making that a reality, and that her life is affected by how safe that food is.

A year ago, almost exactly a year ago, the President made an announcement. This was before he got the healthcare reform; this was before he had figured out where the country was going with respect to Afghanistan. He focused on food safety because he understood it was extraordinarily important. He understood that it was not the kind of issue that was in the headlines every single day, but that every American family was affected, and that this was exactly for what people look to their government, to their public sector; to handle smartly, to handle right, to handle with 21st century tools, and not to neglect.

We all know that that system has to improve. Every year, the statistics -- many of you know them by heart, and of course you're here, in part, to make sure that we're getting this right -- but to the best of our knowledge, one in four Americans, roughly, suffers from foodborne illness. That's a staggering number. And our job here really is to figure out how we can get to a better place.

Our Administration is committed to doing this, not just by trying to use the tools that we already have, although that's an extra-ordinarily important part of the mission. Our Administration is also committed to doing this by pursuing smart, thoughtful, effective legislation on Capitol Hill, which is why we're trying to get a very important food safety bill passed now.

Three principles have emerged to guide the work of the Food Safety Working Group. Number one, preventing harm to consumers is paramount. This is not about recalls first and foremost. It's about making sure we don't get to the point where we have recalls.

Number two, and this is where you come in, good data and analysis are absolutely essential to make sure we have an effective food safety system and to make sure that we have the right balance of different tools and to make sure we're doing enough inspections and enough enforcement.

Number three, when we do have to deal with problems after they emerge, a quick and effective means is required to identify and stop outbreaks of foodborne illness.

To live up to all of these principles, we need to know where we stand. Let me give you, just for a moment, an analogy from the world of tobacco, which many of you are familiar with. In the particular

context of tobacco, each year we lose 5.1 million years of potential life because of smoking, and every single day, approximately 3,600 young people start to smoke. These statistics are important for two separate reasons. Number one, they're important to give us a sense of how much progress we have made, and there are substantial declines in smoking that we need to feel good about. Number two, they're important because they help us understand how much more we need to accomplish. And that's why metrics are so important. Indeed, the successes we have had in the tobacco field owe a great deal to careful, sustained technical attention to getting metrics right.

So I applaud you for convening today, for being here, for bringing your expertise, your knowledge, the perspectives of your organizations. I think diverse representation is absolutely key in this process, making sure that different people from different perspectives were able to be a part of this process. And what you're doing here really is a public forum. It's a recognition that, for all of the extraordinary expertise at USDA, FDA, and CDC, we're still dealing with a domain where no single person and no single agency has all the answers.

I'm very excited about what we can accomplish. The reality is that we need one coordinated effort and that there is no way to make sure that that effort works well if we don't get the metrics right.

Finally, I want to urge you to be ambitious, and let me just underscore why this is particularly important. It is difficult, but not so difficult, to point out how challenging this enterprise is, to figure out why particular metrics don't work, to figure out why the public might misunderstand a particular approach that we take to explaining how we're doing or where we are. I think it's important for us to be honest and to recognize the flaws in any particular system that is proposed to deal with this challenge. But that's not enough. It's extraordinarily important to figure out exactly how we can improve, to hold ourselves accountable, to make sure you push us. I mean, this is a reality, right? We respond to what the American people get concerned about, and we're eager to be intellectually honest and to be clear. But we need to make sure that you hold us accountable, that you tell us, ”This is not the best that you can do. There's actually additional progress that can be achieved.”

So we need the best science to solve the problem. We need the best strategies for engaging the public in this, once we get the science right. I hope you have a tremendously successful meeting, and I hope that you're able to survive in a room like this one, with no windows, on a cloudy and rainy day. But there are few things that are more important than what you're doing here, so thank you for being a part of it.

    (Applause)

MR. TAYLOR: Okay. Thank you, Tino, and, again, thanks so much for being here and the role you're playing. It's really invaluable and appreciated by all of us.

I'm back at the podium. I've got a few minutes allocated to me to just provide some policy context, to provide an understanding from our perspective, at FDA, on why this meeting is so important. And I must say, I share Tino's observation. I mean, to think that on Capitol Hill we've got almost 400 people in a room talking about food safety metrics. For those of us who have labored over this topic and related topics for a long time, it's enormously exciting and gratifying. I think it really stands for the energy that's behind not only progress on food safety, but approaching it and measuring it and doing it in the way that Tino described.

I am particularly pleased that this meeting is being done jointly with our colleagues at CDC and USDA. It is great to be here with Jerry, as well. And I do want to thank all the people who put so much effort into planning this meeting, getting the agenda together and getting us here together in this room.

I think we all know what an opportune time this is for food safety. Tino's talked about it. We have an unusual degree of presidential and Administration commitment to this issue and activity going on. We've got Congress poised to act, to really give FDA a sweeping new mandate, really a new job on food safety. And that's why we're here. We know we're poised for action, and what we're doing here today really matters because of the context in which we're working.

It's also important, I think, and noteworthy, just how broadly shared the vision is of what a food safety system would look like that really can be more effective in reducing foodborne illness. It needs to be, as Tino said, prevention-oriented, and that's sort of the core theme, the message about what I think all of us are trying to do. To reduce the risk of foodborne illness, we have to react, to respond, when problems occur, but prevention is the starting point. That's what public health is really all about. But we also say and believe that this system needs to be science and risk-based, and that sort of gets, in a very large way, to the topics we'll be talking about today. And it needs to be comprehensive. It needs to be farm to table. It really needs to look at risk wherever it can arise. This is really what prevention is all about, taking a comprehensive farm-to-table approach to understanding and seeking to minimize risk.

Now, we at FDA, and certainly our colleagues at USDA, are doing a lot right now, as we anticipate the legislation, to begin to build this prevention-oriented system. At FDA, a good bit of what we're doing is actually planning for the implementation of the new law. We're going to get a mandate for comprehensive preventive controls in food facilities, and so we're planning how that will look. We're actually beginning to develop the regulatory framework for that. We're planning how we're going to use our new import-oversight tools.

We're also looking at how we can revamp our inspection and compliance strategies to implement this new law. We're going to have new tools for inspection, records access, new ways to ensure that companies are meeting their prevention mandate, and so we've got to revamp our inspection and compliance strategy to do that.

We're working to strengthen our collaboration with the states. This is not new, but it's going to be increasingly important going forward. Recognizing that most of the food samples, most of the food inspections, are done by state and local agencies, how do we bring that into the system?

In addition to planning, we're doing some regulatory things. Last summer we issued the new egg-safety rule to address the problem of Salmonella Enteritidis in eggs. That rule becomes effective for the large facilities this July, so we're working on how we're going to implement the new SE rule.

We're actually working, I think as many of you know, on new rules for produce safety; again, beginning rulemaking that we think we could undertake under current law. We'll be able to undertake it much better with the new legislation. But again, we're working hard to address the hazards that arise at that point in the system. And we're investing in scientific and technical capacity to do this new prevention-oriented, science-based job. We've got resources in 2010. The President has requested resources in 2011 that we're investing in building our capacity to implement this new approach to food safety.

And again, the thing to recognize, of course, is that these are inputs. These are means to the end of reducing foodborne illness. And so it's important not just to be busy doing all these things, but it's important, if we're going to be successful, to know, first of all, that we're focusing our efforts on the most significant risks and most significant opportunities to reduce risk, and we've got to be able to measure our progress so that we'll know if we're successful. But equally importantly, we need to learn from our successes and failures as we go down the pathway of seeking to reduce foodborne illness, so that we can continue to improve the system.

And so that's why -- that need to be able to know if we're successful, to be able to measure our progress, to learn from successes and failures -- that's why we're working on the tools that you are going to be hearing a lot about today. You know, we want to be able to systematically, based on data, target our efforts. We've got to understand the burden and distribution of foodborne illness. What are the most significant risks that we need to address? How should we prioritize our research, our standard-setting, our inspection activities, so that we do get the most public-health benefit from our efforts?

We need to measure the effectiveness of specific interventions. We're going to be implementing preventive control rules for Salmonella Enteritidis in eggs. We want to be able to measure the actual progress we make in reducing the illnesses associated with that pathogen. We need to have the measurement instruments to do that.

And finally, we do need to be able to look at the big picture and know that we're making progress in reducing foodborne illness in the aggregate. Are fewer people getting sick? If we don't succeed on that, we won't have succeeded in the fundamental public-health mission that we're setting out to achieve.

So all of these are familiar ideas. I think they're common aspirations. We're all working hard in this direction. But particularly when it comes to this measurement issue, it is far easier said than done, I think all of us agree. And that's really why we're having this meeting today. We really need to, as a community, engage in a dialogue about not only what's the vision, what are some of the things we think we need to do to move toward prevention, but we need to work together on measuring that progress, on being able to evaluate what we're doing to improve what we're doing.

And so at this meeting, you're going to be hearing a lot about work that's going on. And certainly we're going to hear, I think in the first panel, about a lot of work that CDC, in conjunction with colleagues at FDA and USDA, is doing in the epidemiological arena -- looking at the tools that we need to report on aggregate burden of foodborne illness, something that is of great interest to everybody. How do we assess trends in illnesses associated with particular pathogens? How do we determine rates of illnesses attributable to particular foods and particular pathogen/food combinations? Again, this is a critical piece of the information we need to target our efforts and to be able to measure progress on our interventions. We don't regulate diseases. We don't regulate pathogens. We regulate foods. We need to be able to really target our efforts on the foods and the health outcomes that result, so that we can measure progress on prevention.

So all of those epidemiological tools are critical. In fact, we think epidemiology has to play an even greater role than it has played in the past in our overall food safety program.

We're also going to be hearing, though, about some of the non-epidemiological approaches that are being developed, being used in some cases in diverse settings in Federal agencies, State agencies, and in the food industry itself. Contamination data and trends in contamination can tell us something about progress on food safety as an indicator, a surrogate measure, for actual health outcomes. You know, maybe implementation of preventive measures, really knowing how we are being successful in doing the things that we believe will be effective for prevention -- that can be, perhaps, an indicator of progress on food safety -- particular risk factors, behaviors, in retail, for example, that we know are associated with an increased risk, in measuring progress and changing those behaviors. Maybe that's an example of an indicator or measure that can be important. We need to look, from FDA's vantage point, at all of these possible measures, and we really need to look hard at how we can use these effectively. It's easier said than done, because we all know there are real methodological and data challenges in using any of these tools. None of these tools is in a state of sort of perfection at this stage of the game. They're all tools that we need to understand, improve where we can, and consider new tools that meet sort of a basic standard of providing meaningful information. And so, whatever the measure is, it really does have to give us a meaningful indication of progress on food safety. But also, the tools have to be feasible. Again, the methods have to be usable, simple enough so that they can be applied in a routine way. And we have to have the data to apply these methods. And I think data, in particular, is a challenge that we all know needs to be addressed.

There are volumes of data being generated by the food safety system every day, in government, in industry, in the research community. We have no shortage of data in sheer volume. But how do we know we have the right data? How do we get those data together? How do we take advantage of the data that are out there, when they are collected in different ways, by different parties, for different purposes?

And data collection is costly. Again, we have to be really clear that we can devise conceptually elegant methodologies. We can say, “Here are the data that we need to populate them,” and then we can go figure out how we actually pay for the collection of those data. So like everything else, you know, there's no free lunch in metrics for food safety. But we've really got to define where we get the leveraged benefit out of the investments we make, and again, that's why we need this sort of dialogue. That's why we need this conference today. This is really the first of a series of conversations and discussions and workshops that we need to have with the community to address these issues and to really take advantage of the opportunity that we've got now with this unique moment, where we not only have agreement about the general direction we need to move in and, generally, how we can improve food safety, but we've got 300-plus people in the room wanting to work on metrics for measuring progress on food safety. And I just find that an extraordinarily exciting, positive thing.

We very much look forward to working with all of you and to coming up with the metrics that really will be tools we can use to protect public health and to reduce the risk of foodborne illness. So I’m delighted that you're all here. I'm going to turn the podium back over to Nega, who will introduce Jerry, who will also make some opening remarks. And again, I think we all just really look forward to the discussion that follows. Thank you very much.

(Applause)

DR. BERU: Thank you, Mike. Our next speaker in the policy context theme is Jerry Mande. Mr. Mande joined USDA in July 2009 as Acting Deputy Under Secretary for Food Safety. He leads the Food Safety and Inspection Service. Previously, he was Senior Advisor to the Commissioner of the Food and Drug Administration, where he helped shape national policy on food safety. He also has addressed food safety as the health policy advisor on the White House staff and has been Deputy Assistant Secretary for Occupational Health at the Department of Labor. Jerry? Applause)

MR. MANDE: Good morning, and thank you for joining today's metrics workshop. Before my remarks, let me take this opportunity to thank my good friend, Mike Taylor, for his food safety leadership at FDA, as well as the respective staffs at CDC, FSIS, and FDA who helped organize this important meeting.

I also want to thank Tino Cuéllar for joining us in demonstrating what has been an extraordinary commitment by the White House to protecting our nation's food supply. Tino brings a sharp and probing intellect to our task, and we are better for it. There are those here today who have fought on behalf of consumers for years. There are the family members who have lived the horror of our system when it fails. There are industry professionals who have made producing safe food the core of their business, and there are government officials who have been entrusted by the American people to keep our food safe.

We all work toward a common goal: safe food. I want to begin my remarks from that shared premise.

Sometimes we have different ideas of how to get there, what the best policies may be to get us there, but our bottom line is the same: we want the assurance that food won't make our families sick.

Yet providing that insurance requires constant vigilance, and it cannot be done alone. We must work together along the farm-to-table continuum to ensure safe food. The President, Secretaries Vilsack and Sebelius, understand this, and the Administration has sought unprecedented collaboration among its food safety agencies through the Food Safety Working Group and meetings like this one.

The importance of today's meeting can be described in one sentence. What doesn't get measured doesn't get done. I want to state that again, because it is the most important message I want you to take from today's meeting: What doesn't get measured, doesn't get done. Business gurus have repeated this mantra for years, and it is as true for us as it is for them, and it is why this meeting today is so important. Continued progress on food safety depends on adopting and implementing the right metrics.

This is a watershed moment in food safety. These moments, these opportunities, do not come often. But when they do, they significantly change the trajectory of businesses and regulators alike. One came at the turn of the 20th century, when Upton Sinclair's novel, "The Jungle," uncovered the filthy conditions in the meat-packing industry. At that time, foodborne illness was a leading cause of death. Sinclair's exposé, and public uproar that followed, led to the passing of the Food and Drug Act, as well as the Meat Inspection Act, in 1906. We operated under these laws for some time and made steady progress, and for a time, issues such as economic adulteration surpassed microbial adulteration as a major concern, until 1993. It was then that food safety and inspection was turned on its head, once again, when an historic E. coli O157:H7 outbreak in undercooked ground beef caused 400 illnesses and four deaths in the Pacific Northwest. This outbreak was followed by another O157 outbreak, this time in unpasteurized apple juice. In response, USDA, under Mike's leadership, adopted the science-based system of preventive controls we have in place today; the landmark pathogen-reduction, hazard analysis, and critical-control-point system, or HACCP rule, defined a new preventive framework for industry and regulators. It helps protect the nation's food supply through continuous improvement of preventive pathogen control.

With HACCP, FSIS became the public-health regulatory agency it is today, and it worked. We made gains under this system for nearly a decade. We also saw advances during this time period, in the tracking and identification of pathogens, with the establishment of FoodNet and PulseNet, which allowed us to begin linking pathogen reductions to illness reductions.

In 1997, we set a food safety goal for the nation to cut the rates of foodborne illness from the most common pathogens by half by 2010. But we reached a plateau. Most progress toward this goal occurred before 2004. Lower rates mean less illness, so over the last decade, we have moved in the right direction. But we need to know how many people get sick, from which contaminants, in which foods, to design a system that will push through the current plateau and drive foodborne illnesses, hospitalizations, and deaths significantly lower.

When the President took office last year, we were in the midst of a large recall. As you know, and as we've heard, he responded by establishing the Food Safety Working Group within 60 days of taking office, and appointed the Secretaries of Health and Human Services and Agriculture as the co-chairs. Which brings us to this, the next critical juncture in food safety. Guided largely by the working group, we're looking at the entire food safety system and its cross-jurisdictions and products.

It is a watershed moment. We have a President, two Secretaries, and leaders in Congress who have made improving food safety a high priority. The status quo is unacceptable. Our bosses, the American people, have made that clear. In some cases, our laws are outdated, our systems too reactive, and our structure too fragmented, given the complexity of modern food. We need the tools and coordination to meet the challenges of a 21st century food system, and we need better metrics so that we can measure what needs to be done.

Food safety must be improved. Passage of FDA food safety legislation, a high priority for the Administration, is a key step. Today's meeting is another. The Food Safety Working Group has made metrics a cornerstone of our efforts.

USDA is helping lead the change we need to improve food safety. Led by Secretary Tom Vilsack, major new and revamped efforts are underway to improve the safety of the products we regulate. We have challenged our leadership, scientists, and analysts to think strategically and creatively about policies to reduce foodborne illness. We're implementing many priorities identified through the Food Safety Working reduction performance standards and control for Salmonella and Campylobacter. We're actively discussing ways to improve product tracing and better educating and training our work force regarding E. coli O157:H7, and we're supporting the Secretary's renewed emphasis on research, developing new tools, such as a test for non-O157 Shiga toxin E. coli, and promoting food safety research through our new National Institute for Food and Agriculture.

And, of course, we continue preparations to launch our dynamic data analytic system, the Public Health Information System, which will revolutionize the way FSIS detects and responds to foodborne hazards.

But while each of these steps could help bring about the significant reduction of foodborne illness we see, we won't know how best to deploy them unless we can link their use to specific reductions in illness. To do that, we must be able to more precisely measure changes in foodborne illness. And to do that, we must build robust data collection and analysis. We are not regulating for the sake of regulation. We want results.

Policy in any area is best when it's rooted in science and it's measured for impact. The same standard applies to food safety. We want to ensure that our programs and conventions and measures have a positive effect on public health.

So what do we do? What would it take to cut the number of foodborne illnesses in half again? We need assessment tools to guide our efforts, gauge the success of our policies and interventions, and make a direct link between our actions and outcomes. We need to know what is working or not working in order to reach our goal of sharply reducing foodborne illnesses and deaths. And we need specific, measurable, timely markers along the way to know we are on track and to make adjustments if we are not.

For example, although FDA, FSIS, and other agencies have varied roles in our nation's food safety, we essentially begin in the same place: the estimated burden of foodborne illness. Before we make decisions on food safety policies and interventions, we must know how many people are getting sick each year from foodborne contaminants, and which ones. Who, exactly, is getting sick, and from which foods? And, overall, are we making progress toward reducing foodborne illnesses?

These are central questions. However, developing answers to them that we can be confident in and base policy on has proven difficult. To reach our goals, we must measure progress along the entire farm-to-table continuum. Improvements in on-farm interventions can bring improvements at slaughterhouses, which can improve control at processing establishments, and so on, until products reach the consumer.

We must leverage data. FSIS has inspectors in our regulated establishments every day. We must make better use of them to measure and monitor levels of contaminants in our products. And we must better use what we know about industry compliance, process control, and other indicators to assess their impact on public health.

Pathogens evolve and spread through the food system, and as long as we approach them as if they respect the purview or jurisdiction of the farm, the producer, the USDA, FDA, or other agencies, they will elude us. That's why the President has charged us to work in a unified way to meet the challenges of modern food safety system. This workshop is an example of the collaboration our President expects, and what is needed to improve food safety.

In other words, we're in this effort together. The progress of each part of the system is tied to the progress of the other. And we must work together to eliminate foodborne illnesses and deaths. At today's meeting, we'll begin a discussion about reaching this goal in a smart way and measuring our progress on protecting the nation's food supply.

Thank you for being here, and I hope that you will join us to make this a historic turning point in national food safety. On behalf of Secretary Vilsack and USDA, I assure you that we hold ourselves accountable to the public and to making real and considerable gains in improving public health through safe food.

(Applause)

DR. BERU: Thank you, Jerry. I would now like to invite our distinguished speakers to take their seats in the audience.

The next session, "Building a Framework for Improving the Usefulness of Food Safety Metrics," will be moderated by David Goldman. Dr. Goldman is the Assistant Administrator for the Office of Public Health Services at USDA's FSIS. Dr. Goldman was formerly the Director of the Human Health Services Division at FSIS and also has served as Acting Administrator of FSIS. He is a family-practice and preventive-medicine public- health physician and has been a member of the Commissioned Corps of the U.S. Public Health Service assigned to FSIS since February 2002. Dr. Goldman spent 10 years in the U.S. Army Medical Corps practicing family medicine and preventive medicine. He was a District Health Director and Deputy State Epidemiologist at the Virginia Department of Health before joining the Public Health Service at FSIS. David?

DR. GOLDMAN: Thank you very much, Nega, for that introduction. And it's a pleasure for me to moderate this opening technical session for our workshop today.

In just a few minutes, you will hear briefly from three presentatives from each of the three Federal agencies involved in food safety activities, who will describe various frameworks or approaches to measuring the effectiveness of our public-health regulatory programs. Don't be surprised to hear some redundancy, as well as some differences represented in these three talks. The redundancy is the result of our having worked together on this for the better part of a year. The difference, of course, is that each of the three agencies has different needs for the data and the measures that we are trying to create.

The talks will be mostly conceptual. You will certainly hear described the complexities that have already been alluded to in today's opening. But the talks in this session are designed to set up the rest of the meeting, and in the rest of the meeting is where you'll hear more detailed presentations about the use of human health data as a measure of our progress, as well as the use of non-human-illness data as measures of our progress. Very importantly, at the very end of the meeting today, there is an entire hour devoted to you, the public, and our audience, to provide your insights and perspectives regarding how we should best measure our progress.

What I'd like to do is to briefly introduce each of our three speakers so that we can make our way through the transition between the speakers. And I'll also remind you that you will, hopefully, have a few minutes at the end of this presentation for one or two questions, and then we'll go into a break at 10:15.

The first speaker is Malcolm Bertoni, who is the Assistant Commissioner for Planning and Director of the Office of Planning in FDA's Office of the Commissioner. There he directs the agency-wide strategic planning and performance management at FDA and is involved in economic analysis, program evaluation, business process analysis, and risk-communication research. He previously served as the Director of the Planning Staff in the Office of Planning at FDA, and prior to joining FDA, he was Program Director and Senior Research Scientist at RTI International in Washington, D.C.

To Malcolm's right is Carol Maczka, who is the Assistant Administrator for the Office of Data Integration and Food Protection at FSIS, in USDA, where she is integral to the agency's food defense and emergency preparedness efforts and is directly responsible for data modeling, analysis, and integration for the USDA and leads the Data Analysis and Integration Group in that office. Previously, she served as Senior Advisor for Risk Assessment at FSIS, and prior to that, Director of the Risk Assessment Division and Executive Secretary of the National Advisory Committee for Microbiological Criteria for Foods. And prior to joining FSIS, she served as Director for Toxicological and Risk Assessment Programs at the National Academies of Science Board on Environmental Studies and Toxicology.

And to Carol's right is Chris Braden, who is a medical epidemiologist at the Centers for Disease Control and Prevention, where he serves currently as the Acting Director for the Division of Foodborne, Bacterial, and Mycotic Diseases. Previously, Dr. Braden served as the Associate Director for Science in the Division of Parasitic Diseases, and before that, as Chief of Outbreak Response and Surveillance within the Enteric Diseases Epidemiology Branch. Dr. Braden also served as medical epidemiologist in the Division of Tuberculosis Elimination. He joined CDC in 1993, as an epidemic intelligence officer, and is a Captain in the Commissioned Corps of the U.S. Public Health Service. Malcolm?

MR. BERTONI: Thank you, Dr. Goldman, and good morning, everyone. I'm going to give some remarks here today that are going to provide a bridge from the policy context that you've heard about already to some of the science and epidemiology that you'll be hearing about later in the morning and this afternoon.

So this framing is really intended to talk about how we can improve our nation's food safety system by improving our ability to make risk-informed decisions. This is an overview of some of the points I'll be making briefly in this first talk.

Better food safety decisions require improvements in at least three areas: better metrics, better targeting, and better data. And here's what I mean by that. For better metrics, we need to expand and improve our measures of how the food safety system performs. And in particular, we need to improve the measures that help us better understand how our hazard process controls relate to changes and outcomes, such as reductions in foodborne illnesses and deaths.

The Federal government has taken a good step in this direction with the development of some consensus high-level metrics that we'll be discussing briefly here. Now, also I want to note that there will be a lot of details in these slides. You don't have the slides with you. We can make those available to you later.

Secondly, for better targeting, we need to improve the methods that we use to allocate resources so that we get the greatest public health benefit possible from our investments. Several speakers this morning will be addressing some analytical tools and methods that move us closer to this goal.

Third, for better data, we really need to improve the quality, timeliness, and completeness of data feeding into the food safety measurement system. Many of the speakers today will be discussing efforts to advance this through improved science, technology, and management.

So, as you've already heard, for the past year, FDA, CDC, and FSIS have been collaborating on a number of efforts to coordinate measures for the food safety system. These measures are very high-level measures, and if you look at them, you'll see that a lot more detail needs to be worked out in particular cases, to identify particular food/pathogen or food/toxin combinations that could actually form the basis for a data collection program. Some of these are already in place. Others need to be put in place.

On the left-hand side of this slide, you'll see the -- and also, let me say, as well, I'm going to show you a sequence of three slides that are organized around those three principles that Dr. Cuéllar talked about. First is prevention. And here on the left-hand side, you'll see a measure related to hazards that are out there. This one, in particular says, “Prevalence of selected foodborne hazards in key food commodity groups.” So clearly, we need more specificity about what we're measuring there. But that gives you a sense of what bad is out there that could potentially cause harm.

The middle section here is talking about a very important area, and that is the measures of preventive controls. The three that we've noted here are percent of food facilities with effective preventive controls; percent of retail and food services establishments with adequate controls; and proportion of consumers who follow key food safety practices. Again, more details are needed, and you'll hear more about that later today.

But, really, the measures on the right-hand side really get to the heart of the matter. Those are the public health outcomes; incidence of foodborne illnesses; number of outbreaks and associated cases; number of severe illnesses from food allergies. As you've already heard from all three previous speakers earlier this morning, this link between the controls and the outcomes is really the key. And that's what we're going to talk about in a few minutes.

On the next slide, we have a group of metrics that are associated with the second core principle of the Food Safety Working Group. This is addressing surveillance, risk analysis, inspection, and enforcement. So there are a number of measures here. Again, I'm not going to read through all of these details, but you can get a sense that when we're talking about risk-based targeting, we're looking at, say, our ability to measure the number and percent of foodborne illnesses attributed to specific food commodity types. And CDC is going to be talking about improvements to our ability to make those attributions.

And then, of course, there's the important issue of the inspection and surveillance. So how are we overseeing the food supply chain? And there are a number of measures here that we'll be continuing to improve over time.

And of course, there are also measures of enforcement. Now, as Mike and others mentioned, measurement in this area is very challenging. You've already heard the old saying that what doesn't get measured doesn't get done. I've heard it the other way around: you get what you measure. So you can imagine how careful one must be when constructing a measure of enforcement, so that you maintain a focus on public health and ensure appropriate incentives for fair and impartial treatment. You know, we wouldn't create a system that's the equivalent of a traffic cop with a speeding-ticket quota. So this is a challenging area that we're working on, and other speakers are going to address these in more detail as the day goes on.

On the next slide, we have a third set of measures that are associated with the third core principle -- response and recovery. Of course, these measures address activities that come into play when the controls fail and contaminated food enters commerce and causes harm. Just as an example, you'll hear more from the CDC about the first measure here, on the average number of days to identify sub-types for priority pathogens in humans and food – critically important for really tracing through what's going on in the system.

In the middle of this list, there is the annual average number of days from identification and trace-back of implicated product to the initiation of a recall or equivalent protective action. So there we have a real joint effort between all the three agencies that are here today and working together to improve our performance in that area.

Then there's also a measure on effectiveness of food safety recalls, as measured by annual average percent of recalled product available in commerce a specified number of days following recall initiation. How well are we getting bad stuff off the shelves, how quickly? So these are things that we'd rather not have to do, but if the other preventive and oversight controls fail, this is something that we need to do.
Now, you'll also notice that most of these particular measures are really -- if you look at them closely, later, when you get the details -- you'll see that they really focus primarily on response. And to the extent that we do a good job with response to an outbreak or other contamination event, then the recovery effort certainly will be easier. But that said, I do believe we need to recognize the importance of quick recovery and develop better measures in that area, as well. So again, you'll hear a lot more about some of these measures later on in the day.

On the next slide, I'd like to change the focus a little bit to how are we going to do a better job of targeting our investments to improve the food safety system? Certainly, measurement systems are expensive, and we can't measure everything. So we need to make sure we're getting the best return on our investments.

I'm going to talk very briefly about three different approaches that we're already using and that we need to perhaps do more of, do a better job of, in the future, in order to bridge that gap between the science of what's happening out there and the interventions that we in the Federal government are responsible for leading and working with our State and local colleagues to execute.

So the first of the three is what I'll call logic modeling. Now, this is a technique for identifying logical relationships between the public health outcomes we're trying to improve and the chain of activities, outputs, and intermediate outcomes that cause or contribute to those public health outcomes of interest.

On the next slide, I provide a little graphic example. It's actually not directly from the food safety arena -- I don't want to prejudice people's thinking here. But you can illustrate the concepts. And on the left-hand side, what you have are the activities and outputs of a particular organization. In this example, it's FDA's activities related to food and nutrition that ultimately relate to trying to improve a public health outcome of reducing deaths from coronary heart disease. And what this kind of approach does is that it helps you understand the logical relationships between, and the chain of causation that we hypothesize, between these activities, regulatory processes that produce outputs, such as nutrition facts label requirements -- everyone's seen those; consumer nutrition and health information that may be on the FDA website, for example. And hopefully, that is having an impact out in the world, such as nutritional knowledge. And we do surveys to look at the percentage of American consumers who correctly identify that trans fat and saturated fat increase the risk of heart disease. Now, hopefully that knowledge will actually have an impact on eating behaviors, but I think there is a lot of research to be done about what other factors affect it in addition to knowledge. But the eating behaviors are another intermediate outcome. How much trans fat and saturated fat is actually consumed by people? And ultimately that, along with a whole other host of things that FDA does not control that are shown sort of in the blue areas here, such as total caloric intake, smoking, exercise, genetics, incidence of obesity, medical treatment -- all of those things contribute to coronary heart disease.

Now, this kind of analysis of all the different contributing factors that an agency controls and doesn't control, I think, is very important for making sure that we're measuring the right things, and that we're keeping our eye on the ball, keeping our eye on the public health outcomes and the intermediate outcomes that are having a positive effect in the world. And if the actions that we're taking are not having the intended effect, we need to analyze and understand why.

So in the next slide, I want to highlight another tool that we're using to try to target our resources and understand where best to measure. I've called it supply-chain modeling here. It's basically a process-modeling approach, almost like an engineering approach, to understanding the sequenced network of activities that are involved from bringing food from the farm to the fork.

Now, on the next slide, there's again a high-level model that gives you a sense of the kinds of stocks and flows that are involved in bringing food to market. This is a very high-level model, and to be useful in a particular study, or, for example, a response activity, you would need to, for example, support trace-back, you would really need to identify the exact details of which foods were involved and which particular firms are involved. You would probably do that through database modeling; which firms are in your database, and you'd have to look at quantitative information about the stocks and flows. And this is done in particular areas, and building better models of how the food system works is part of getting a better handle on where we need to measure and how we can better understand what's going on. And you'll hear a little bit more about that in Carol's talk in a very specific example.

And on the next slide, I'm going to make a brief pitch to bring in more of the tools of decision analysis to this process, as well. The field of decision analysis focuses on rigorous science-based treatment of how one should choose among alternative courses of action in the face of uncertainty. It's built on mathematical and statistical sciences, psychology, and economics, and some of those fields. It's inherently an interdisciplinary field. And it really addresses precisely the kinds of problems that we're dealing with today -- how to invest our limited resources in building a stronger food safety system.

And in particular, there is a methodology regarding analysis of the value of information, where you're looking at how imperfect and incomplete data come together and where you should be investing and when you should be investing in more information. And I think that those are important analytical tools that we need to bring to bear more than we have in the past, so I do encourage the academics and practitioners in the decision-analysis field to apply their talents to addressing this challenge.

So in the remaining slides that we'll make available to you, I've provided a little bit more information to explain some of the terminology that we've used here. In the next slide, you can see generic types of measures about inputs, activities, outputs, intermediate outcomes, and such. And there's also, on the next slide, some ideas for essential criteria, for good measures.

And on the final slide, looking at the data, some data quality indicators that are important when we're designing data collection systems.

And again, those slides will be made available to you and the rest of the folks here.

So that concludes my remarks. Thank you very much for taking the time to participate in today's workshop, and I'm looking forward to the dialogue.

(Applause)

DR. GOLDMAN: Thank you very much, Malcolm, that was a very good overview of the concepts that we need to keep in mind as we develop performance measures.

We'll next hear from Carol Maczka. DR. MACZKA: I'm going to take it from my chair here today. I'm nursing an injury.

Okay, so what I'd like to talk about is a farm-to-table approach for food safety metrics. And I'm going to touch on the interagency food safety metrics; then I'll present the farm-to-table framework for performance measurement, and particularly as it relates to the coordination between FDA and FSIS on Salmonella Enteritidis reduction. And then I'll talk very briefly about performance measurement at FSIS.

So -- next slide. And the next. Okay.

As the previous speaker stated, collaboration was, and continues to be, a key aspect of the Food Safety Working Group. And when Jerry Mande spoke, he emphasized the need to work together along the farm-to-table continuum to produce safe food. He also emphasized the need to have goals that are smart, specific, measurable, attainable, relevant, and timely. And so, under the auspices of the White House, FDA and FSIS and CDC work jointly to develop food safety metrics. Next?

The interagency food safety metrics. Some of these metrics are final public health outcomes. An example is, as Malcolm presented, the number and percent of foodborne illnesses attributed to specific food commodity groups, another example being the number and percent of foodborne illnesses attributed to a particular pathogen. Next?

Other metrics, though, are more interim in nature. And what they do is assess the agency's activities further up the supply chain. These activities are intended to reduce foodborne illness. They are intended to address the final outcome. Examples of these types of interim measures are the prevalence of selected foodborne hazards on key commodity groups, another example being percent of food facilities with effective food safety controls. Next?

In addition to developing final and interim metrics, the agencies also agree that these metrics should be applied at targeted points along the farm-to-table continuum. And this is because food production can be described as a system comprised of multiple steps under the regulatory purview of several agencies, for example, FSIS, FDA, and APHIS. Next slide? The advantages of looking at farm-to-table in this framework is that they demonstrate the need for a coordinated approach to ensure food safety; they help identify data gaps to measure the impact of food safety activities; and they help to target areas in the farm-to-table continuum where more attention is needed. And I'm going to demonstrate this using Salmonella Enteritidis. Next slide?

Okay. So the agencies agreed to work together to decrease foodborne illness from Salmonella Enteritidis. FDA, FSIS, and AFIS have programs in place to target SE reduction in poultry, shell eggs, and processed egg products. And to measure the impact of these programs on Salmonella Enteritidis reduction, metrics were developed along the farm-to-table continuum.

So this next slide shows the regulatory jurisdictions of the agencies with respect to poultry, processed egg products, and shell eggs. And this is broken down by on the farm, slaughter and processing, and distribution, retail, and home. And it acknowledges that throughout the supply chain, transportation occurs and is under the jurisdiction of FDA and FSIS.

So if you look on the farm, FDA and APHIS have jurisdiction over chickens. And some of those chickens go on to be layers and produce shell eggs. The layers are under the jurisdiction of FDA and APHIS; the shell eggs, FDA. Some of those chickens go on to be broilers under the jurisdiction of FDA and APHIS. The shell eggs, if you move on to slaughter and processing, can be broken or sorted. And the breaking of the shell eggs results in processed egg products. The shell eggs are under the jurisdiction of FDA; the processed egg products, FSIS. Those products then move on to distribution centers, retail, and institutions.

The broilers are presented to slaughter, and they become intact poultry or ground poultry. And those products are under the jurisdiction of FSIS, and again, they move to distribution centers, retail, and institutions.

So using this process-flow diagram, the agencies then develop metrics along the continuum. So if we move to the next slide, this is illustrative. It's not meant to be an inclusive list of all the metrics and measures we develop. But what I'm trying to show here are, first of all, the steps in the farm-to-table continuum. And you can see we move from production and harvesting through slaughter and processing, distribution, retail and food service, home storage, and then public health surveillance and outbreak response.

And what I've provided here, then, are examples of metrics and measures along targeted points in the supply chain. So if you look at production and harvesting, you see the metric, prevalence of selected foodborne hazards in key commodity groups. The FDA measure, proposed measure, would be the percent of expected laying houses that tested posit live for SE during environmental monitoring. Another metric proposed is the percent of food facilities with effective preventive controls, the FDA measure being the percent of violative laying houses.

If you look at slaughter and processing, you see the metric once again: prevalence of selected foodborne hazards in key commodity groups. Here, we have an FSIS measure, the percent positive rate for Salmonella Enteritidis in FSIS pasteurized egg products, broiler carcasses, broiler parts, ground chicken, and RTE products.

If you move on to retail, once again we see the metric, prevalence of selected foodborne hazards in key commodity groups, with the FSIS measure being the percent positive rate of Salmonella Enteritidis on food products at retail. And we also included a metric that you saw under production and harvesting, the percent of food facilities with effective preventive controls, with the FDA measure being refrigeration at retail.

And then finally, if you move to the public health surveillance and outbreak response, the metric you see there is the number and percent of foodborne illnesses attributed to specific food pathogens and/or commodity groups. And here, FSIS and FDA agreed on a common measure, and that is, by the end of 2011, we would decrease by 10 percent the rate of sporadic Salmonella Enteritidis illnesses.

So let me stand back and make a couple of points about this table. First of all, again, these are proposed metrics and measures. They are not intended to be inclusive. And in some cases, we're not currently collecting data for some of the measures, an example being under retail, the FSIS measure, the percent positive rate of Salmonella Enteritidis on food products at retail. We currently do not collect that information, that data. But if we knew that the Salmonella Enteritidis illnesses were increasing, and if we were to look at slaughter and processing and we were to note or have data that the percent positive rate for SE was decreasing in FSIS-regulated products coming out of slaughter and processing, we might very well need to look at retail and determine what the percent positive rate of Salmonella Enteritidis is on food products at retail, because it may be there that the problem actually resides. So the metrics and measures that I presented in this table, I presented them because I wanted to make the point that if we measure prevalence and controls along targeted points in the food supply, that we may be able to actually identify the source of the problem. Is it shell eggs, is it processed eggs? Is it laying houses, or is it at processing? Is it occurring at processing, or is it occurring at retail?

So the benefits of looking at things according to this framework is that it helps to tell a story about food safety. It helps you identify data gaps. And it also helps you to identify where resources need to be targeted. Next slide?

Now, once we've developed these metrics and measures, this is what we intend to do with them. So for each measure, we would establish a baseline based on data. And I didn't fill anything in here. This is really just to illustrate what we would do.

Once we established our baseline, then we would establish yearly objectives, with the end goal being 2015. I say 2015 because that links to FSIS' strategic plan. And then quarterly, what we would dois measure the progress against the objectives. And if we find that we're not meeting our objectives, then what we would want to do is put corrective actions in place or additional policies and practices.

This next slide is just to say that the three agencies are working together to develop measures related to SE for additional metrics. And the next slide shows some of the other metrics under consideration. I'm not going to go into these in detail. You'll have them in your information packages, and Malcolm already did touch upon some of them.

So I'm going to end with this final slide, which is just to touch on how at FSIS we're measuring performance. And over the last two years, we've developed performance goals and objectives that allow us to measure our progress toward reducing foodborne illness in our regulated products. I'm not going to go into the rest of these slides, because later on today, at 2:00 o'clock, we will present how we use foodborne illness attribution data and pathogen verification testing data to develop performance goals, objectives, and measures.

So I'm going to end here, and hand it over to Chris. (Applause) DR. GOLDMAN: Thank you, Carol, and thank you for that example of how important it is to measure hazards upstream in the production process. And we'll hear more about that, as you said, later in the day. And the last panelist here is Chris Braden.

DR. BRADEN: Thank you, David, and good morning, everybody. My name is Christopher Braden, and I'm from the Centers for Disease Control and Prevention. And I'm going to be presenting to you yet another graphic about a framework for metrics and foodborne diseases that has, you'll see, some significant overlap with what you've heard before. Next slide, please.

So this is a CDC-centric view, I think, what we call the Public Health Life Cycle. And for us, this is how we do a lot of our business, in starting at the top with surveillance. Surveillance will lead to investigations, when there's a problem identified. Those investigations can lead to some acute interventions to ameliorate the problem, or in addition, it may be that there's some applied research that's needed in order to identify the interventions that are necessary. But those interventions are then measured, again, with surveillance and monitoring function.

So I'm just going to go through some of these steps to give you an idea of what's involved, again mainly from the CDC's perspective, but it touches all the agencies. And not go into specifics about metrics, but I may mention a few of them along the way. So next slide, please? Thank you.

So this is what we call an epidemiologic curve, the number of cases over time depicting an outbreak. This outbreak is the Salmonella Tennessee infections associated with peanut butter in 2006 and 2007. You can see the arrow there, when surveillance really first identified this cluster for us. And many of you may be aware of the consequence of the investigation of this outbreak. But this is just to illustrate the fact that this is an obvious example of an event that was detected through surveillance.

But surveillance depends on a number of factors. Next slide, please.

This is the number of uploads to PulseNet, one of our premier surveillance systems for detecting outbreaks. PulseNet collects what we call pathogen subtypes, that we are able to link cases together with, that indicate a potential outbreak. And you can see that, over the years, PulseNet has matured, starting from relatively few patterns uploaded to the PulseNet national database in 1996 and 1997, to about 60,000 images uploaded that would indicate the genotype or the DNA fingerprints for those pathogens in 2007-2008. It remains at about that level. But again, here we have, what is the completeness of your data? Especially when you talk about 60,000 pathogens versus what we know about the hundreds of thousands of illnesses that occur every year, just as Salmonella alone.

Next slide, please. This slide relates to the time limits of the data. And you can see if you add up the times between when a food is contaminated; when a patient eats it and becomes ill; the stool sample is collected; the Salmonella is identified; the public health laboratory receives the specimen, etc., that you can be looking at between two and three weeks before the public health system knows the specifics about this particular agent. Some of these things we have control over; some of them we don't. It would be hard, I think, to have control over the incubation period of our pathogens or the healthcare-seeking behavior of our patients. But we can increase the timeliness when it comes to getting isolates where they need to be and doing the tests that we need to do. Next slide, please. So going on to outbreak investigation, this slide represents, on a very simplistic level, one of the pieces of investigation that has to do with the epidemiologic studies we do to identify food vehicles. These circles represent people. The red ones represent ill people. What we have to do as investigators in this particular type of study is to interview each one of these patients. Now, remember, we're interviewing them two to three weeks after the onset of illness. They may be hard to reach. They may have difficulty identifying what they did or did not eat the week prior to their illness. In this particular example, for instance, we're asking about tomatoes. But that's just one of potentially hundreds of exposures that we may ask every individual about. So these are difficult investigations just in the fact that it takes a long time to reach these people to interview them and get this exposure history. In this particular circumstance, we've seen that six of 11 of the ill people were exposed to tomatoes. Three of seven not-ill people were exposed to tomatoes. We would say that that odds ratio, given that information, is probably not suspect.

Okay. We have to go on and identify the other information that we have that would potentially lead to some hypothesis as to the result of the outbreak. So next slide, please?

Here we come to the applied research section. Applied research is a very broad field. This gets to the root-cause analysis of what may have resulted in a foodborne outbreak. In this particular circumstance, this water may be used for agricultural purposes. We may want to know more about the water quality used in agriculture, if that is the source. So that is just one example of the myriad of issues that may be investigated, as far as applied research in food safety. And there are probably some good measures that we need to address having to do with applied research. How often do we actually do this root-cause analysis to find how the product originally became contaminated, and how often are we successful when we do embark on that kind of effort?

Next slide? So then, coming to interventions. You'll hear a lot more about intentional interventions. I've listed just a few. They're not supposed to be complete, but certainly there are some acute interventions, like product recall and acute production process correction. There are more medium and long-term types of interventions, like systemic production process correction, technological advances that may be implemented, and of course regulatory adaptation.

There are certainly some metrics that we could apply to some of these. One of the easiest ones may be -- it's not easy, but one that's identified most often -- may be the effectiveness of recalls. But how often are we able to actually go on to determine some of these intermediate and long-term types of interventions may also be helpful measures, though difficult to measure?

Next slide? So here we come to the monitoring of what our interventions are able to do. You will see this graphic again today. These are the trends of illnesses due to specific pathogens over time. And this is the type of data that surveillance can provide to lead our efforts. I won't go into the details of this; you'll hear more later.

Next slide, please. So here's our Life Cycle of Public Health again, and this time with some of the metrics indicated on the cycle. And of course, there's surveillance going off clockwise. We have completeness and timeliness for investigation. You have how many of those investigations actually identify a food vehicle for applied research. You may have what are root causes of food contamination identified for interventions. Are there effective interventions applied, and how quickly are they applied? And then, going back, tying back into our surveillance system would be monitoring trends in foodborne illness.

I would say that the first three-quarters of that clockwise rotation could be considered process measures, and then this last quarter-hour between intervention and surveillance would be the outcome measures.

Next slide? So for process measures, there are actually quite a few that have been identified, probably more that will be identified, some of them coming from the President's Food Safety Work Group. You can read a couple of them there. But also in the Council to Improve Foodborne Outbreak and Response publication; these are the guidelines for foodborne outbreak investigation and surveillance, and they have a whole section on measures that mostly revolve around process. And I would say that process measures may be some of the easier ones to try and identify in our food safety system. Next?

But what we'll be talking quite a lot today is impact measures, that last part of relating our interventions back into the surveillance circle. There are a lot of challenges regarding impact measures. Some of them are just exemplified here. It’s certainly not a complete listing, but pathogens, remember, may be transmitted by routes other than food. So what are you really measuring when you are using metrics for food safety, when, actually, the transmission could have been directly from environment, or person to person, or animal contact? All of these pathogens can be transmitted, to varying degrees, by these other routes of transmission.

Also, illnesses may be associated with a myriad of pathogen/food combinations, so which are the right ones to measure, and do we have enough information on any one pathogen/food/commodity combination to really make an informed decision?

Also, data sources are limited, and not specific. For instance, if we just look at case reporting through our surveillance systems, case reports are not linked to the food vehicle. So that particular fact limits how we can use surveillance data alone for cases in identifying how they are affected by interventions in food safety. Outbreaks are associated with food vehicles, but outbreak cases account for only a small proportion of all foodborne illnesses. So if we're depending upon the outbreaks that are linked to food vehicles, but that's a very small proportion of cases, is that proportion of cases representative of all the cases that are out there? And this is an open question, I think, that we need to assess better. Also, there are a lot of unknown pathogens and unknown food vehicles. So even if we target our interventions for specific things about what we know, we may be measuring a lot about what we don't know. And that will muddy the waters.

So these types of issues will be the focus of much of the discussion today, and we'll hear about them from a number of presenters from CDC.

Thnk you very much for your attention. (Applause) DR. GOLDMAN: Thank you, Chris. Thanks for providing us some insight into the work that CDC is doing to fully characterize the human illnesses related to foodborne causes that will help the regulatory agencies apply their efforts in a more precise way.

Also, please help me thank all of our panelists for this great opening. (Applause)

DR. GOLDMAN: We are a bit overtime, and what I would like to suggest is that if you have questions for any of the three speakers, that you try to seek hem out during the break or save it for the afternoon, when we will have some time -- quite a bit of time -- devoted to questions and, hopefully, answers, as well, that have been generated throughout the day.

We do have a break now, and I want to have everyone take their 15-minute break, because there are a lot of people in the room. And we'll see you back here at 10:45. Thank you. (Break)

DR. BRADEN: If people could come in and be seated, please, we'll start our next session.

We're starting our next session now, if people would come in and be seated.

Good morning again. I'm Chris Braden. I'll be moderating this session on the Measures of Progress Based Primarily on Human Data. This session is a nice segue from the last points that I was making in the previous talk that I gave, whereby I outlined some of the challenges having to do with determining the impact of our interventions using some of the data that we have available to us. And a lot of that having to do with human illness data is derived from surveillance, surveys, and studies that are done at the Centers for Disease Control and Prevention. We have a nice set of talks for you today that go over a number of the measures that we either have produced or are producing or will produce, including some of the trends, the burden, attribution; you will know those terms well by the end of this session.

So we will try and save some time for some Q&A at the end of this session, but I anticipate that it won't be enough time to answer your questions. So please remember to write your questions down so that at the end of the afternoon, when there's an open-mike period, you'll be able to get your questions answered then, or grab a participant in the meantime.

I'll be introducing each of the speakers before their presentations. The first speaker is Dr.Olga Henao. She'll be giving a talk on estimating and tracking changes in the incidence of illness due to major foodborne pathogens using FoodNet surveillance.

Dr. Henao is an epidemiologist in CDC's Foodborne Disease Active Surveillance Network, otherwise known as FoodNet. Dr. Henao leads efforts within FoodNet to conduct, explore, and implement enhancements to analysis of surveillance and study data. In her current position, she guides the development and enhancement of statistical methods and techniques used to examine trends and incidents of foodborne illness over time. She also leads a work group in charge of development, implementation, and analysis of a national population survey conducted in burden of foodborne disease estimates. Please welcome Dr. Henao.(Applause)

DR. HANAO: Thank you, Chris, and good morning, everyone. As Chris mentioned, I will be talking about estimating and tracking changes in the incidence of illness due to major foodborne pathogens using FoodNet surveillance.

And to start out, let me give you background on what FoodNet is. FoodNet is the Foodborne Diseases Active Surveillance Network. We were established in 1996 and serve as the principal foodborne disease component for the Emerging Infections program at the Centers for Disease Control and Prevention. However, FoodNet is really a collaborative effort of the Centers for Disease Control and Prevention, the U.S. Department of Agriculture, specifically the Food Safety and Inspection Service, the Food and Drug Administration, and 10 participating State health departments.
The States participating in FoodNet are shown here. And going from west to east, consist of the full States of Oregon, New Mexico, Minnesota, Tennessee, Georgia, Maryland, and Connecticut, and selected counties within California, Colorado, and New York.

The surveillance area covered by FoodNet covers approximately 46 million people, or about 15 percent of the U.S. population.

Within FoodNet, we have four main objectives that we work under: We want to determine the burden of foodborne illness in the United States; we want to monitor the trends and the burden of specific foodborne illnesses over time; we want to be able to attribute that burden of foodborne illness to specific foods and settings; and use this information to disseminate information that can lead to improvements in public health practice and the development of interventions to reduce the burden of foodborne illness.

In the rest of my talk, I'll be concentrating specifically on how we monitor trends in the burden of specific foodborne illnesses over time. Now, one thing, in order to monitor trends, we have to have accurate data. And the data that we have is collected via active surveillance for laboratory-confirmed infections of Salmonella; Shigella; Campylobacter; Shiga toxin-producing Escherichia coli, otherwise known as STE;, Listeria; Yersinia; Vibrio; Cryptosporidium; and Cyclospora. We also conduct surveillance for hemolytic uremic syndrome among a network of pediatric nephrologists, using hospital discharge data review.

Using this surveillance data that we collect, we analyze it, and every year produce a report on the incidence of infection with pathogens commonly transmitted via food. This report is published every April in the Morbidity and Mortality Weekly Report, and over the years has become a sort of report card on the food safety system. This document and this analysis is used by regulatory agencies, industry, and public health personnel to prioritize and evaluate interventions, and also monitor progress toward the national health objectives.

So what are the methods that we use in order to analyze our data? First, we look at the incidence of new infections that occur within our general catchment area. And we calculate the incidence, or the number of new infections per 100,000 population. To calculate this, we divide the number of laboratory-confirmed infections by U.S. census population estimates, and then multiply by 100,000 in order to have a metric by which we can compare this information among sites, among different pathogens, and so on.

Using this incidence information, we can look at a variety of things. One example is monitoring progress toward Healthy People 2010 objectives. In this chart, you see the pathogens for which we have health objectives, the 2010 health objective per 100,000 population, and, in the last column, the 2008 incidence for each of these pathogens. And we're able to, each year, see what type of progress we're making toward those objectives.

What we found out from 2008 data is that we made a good bit of progress with respect to STEC 0157, Camplyobacter, and Listeria, yet for Salmonella, we're still pretty far away from that objective.

We're also able to use our incidence information to look at what is occurring within our sites. We know that incidence rates vary by site, and they vary quite dramatically. In this chart – which is a little hard to read, but will be in the materials that you receive from the presentations in this meeting -- if you look at this chart closely, you'll see that, for 2008, the incidence, for example, of Camplyobacter range from a low of seven cases per 100,000 in Georgia to a high of 30 cases per 100,000 in California.

For Yersinia, in contrast, we saw a low of one case per 100,000 in Colorado to a high of 2.6 cases per 100,000 in Georgia. So we know that there is a good bit of variability from site to site with regard to incidence.

We're also able to look at our information in a variety of subgroups, and that includes age group. And this is just one example of the looks that we can take at our data. Here, we have five age groups that we've looked at: under four years of age, four to 11, 12 to 19 years of age, 20 to 49, and 50-plus. And we can look at them in many other ways, but here what we find, as a highlight, is that we see, generally, very high incidence rates in the youngest group, the under four. Here we see the highest rate for Campylobacter, Salmonella, Shigella, STEC O157 and also STEC non-O157, and Yersinia. And for some pathogens, we see the highest incidence in the oldest age group, such as for Vibrio.

Now, in addition to looking at incidence, that number of new infections per 100,000 population, we also want to try to understand how the patterns of incidence have been changing over time. And for that, we have one method that we use. And we use that method in order to compare what's been happening in any one year with the first three years of surveillance and also with the preceding three years, so that we have a comparison of what happened since we first started the program and what's been happening recently.

Now, in looking at change in incidence over time, one of the things that we have to take into account is that the surveillance area covered by FoodNet has not always been the 10 sites that I listed before. In fact, we started with five sites in 1996, and since then we have grown to the 10 sites that we cover now.

And because we've had that change in surveillance area over time, what we do to estimate change is model the change over time using a negative binominal regression model. This allows us to account for that increase in number of participating sites, and then, also, that site-to-site variation in incidence that I pointed out before. What this model allows us to do is to calculate a relative rate of change: a percent change and a 95 percent confidence limit. So we can tell you what type of change we've seen compared with the first three years of surveillance; what type of change we've seen compared with the preceding three years; and, also, if that change is statistically significant or not.

The way that we traditionally present this information in the different reports that we produce is using the graph that Chris showed you earlier. Here,on the X axis you have the years and on the Y axis you have the relative rates. And then the middle line that you see, right here in the middle, is at one, relative rate of one, which means there is no change in the rate.

What we do is, we look at the rate for an individual year, compare it with the rate for the first three years of surveillance, and determine how it has changed. Anything above the line indicates an increase in the rate. Anything below indicates a decrease.

So if we look at this chart, for example, and look at Vibrio, which is denoted by the solid brown line, what we find is that, since 2001-2002, the rates of Vibrio for each year have been consistently higher than the rate in the first three years of surveillance. Conversely, if we look at the line -- the dashed green line -- for Campylobacter, what we find is that the rates of Campylobacter for each year have been consistently lower than in the first three years of surveillance.

We also know it's important to look at what has been happening recently, because this graph concentrates on what's happened each year compared with when we first started. And to do that, we have another type of graphic that we use. Here you see on the X axis the different pathogens that we have estimates for. On the Y axis, you have percent change. The center line indicates no change. Then we have a dot that indicates the percent -- the estimate of the percent change. And then one thing that we do in this graph that we don't do in the previous one, but that I alluded to before, is that we include the 95 percent confidence intervals, so that we're able to see what range that estimate could take on.

What we see in this graph, for example -- this is for 2008 compared with the preceding three years, which would be 2005, 2006, and 2007 -- and what we found in 2008 when we looked at our information is that whereas we saw changes in 2008 compared -- when we compared it with the 1996-1998 period, when we looked at this more recent time period comparison, we found there, in fact, was very little change in the incidence of 2008 compared with those preceding three years.

And those are just some of the examples of things that we are doing. Future work in the area of trends in the incidence of foodborne illnesses within FoodNet include examination of trends by age group. And not necessarily just restricted to those age groups I showed before, but really taking into account the different -- the inputs that we received regarding what age groups are important to look at.

We feel it's important to determine whether the different trends are at play in different segments of the population in order to really guide interventions that are put into place. Also, analysis of trend and outbreak-related cases. What effect are those having on the trends that we're seeing overall? This could provide additional insight relevant to prevention efforts.

Also, we're working to expand modeling approaches to adjust for seasonality. We know that our pathogens, the incidence of pathogens, do have seasonality associated with them, and this will allow us increased precision in the estimates that we produce.

And finally, we're looking at the calculation of summary measures of change, an examination of the data to detect natural breakpoints in incidence rates over extended periods of time. We have quite a number of years of data now, and so one of the things that we want to be able to do is try to understand how those changes are occurring within specific time periods. And with that, I would like to thank you. (Applause)DR. BRADEN: Thank you, Dr. Henao.

I'd like to introduce next Dr. Elaine Scallan. She's going to be presenting a talk on methods for estimating the total burden of domestically acquired foodborne illness.

Dr. Scallan is an Assistant Professor in the Department of Epidemiology at the Colorado School of Public Health, where her research focuses on the burden and attribution of foodborne diseases. She previously served as the team lead for FoodNet at the Centers for Disease Control and Prevention. Please welcome Dr. Scallan. DR. SCALLAN: Good morning. So this morning, I'm going to give you a bird's-eye view of some of the methods we're using to estimate the burden of domestically acquired foodborne illness in the U.S.

I don't think I need to convince anybody here why it's important to measure the overall burden of disease. Periodic assessments of the total burden of illness are an important contribution to the metrics we're talking about today in helping us to effectively set interventions aimed at reducing the burden of foodborne disease.

These burden estimates also lay the foundation and inform some of the other things we're going to talk about later, like attribution or economic assessments, so there they have a contribution also.

There are challenges to estimating the overall burden of disease, though. We've already heard a little bit about them in the previous session, but just to reiterate some of the key points. First, only a small fraction of illnesses are actually confirmed by laboratory testing and reported to public health agencies. Chris showed the diagram of how long each of these steps take for an illness to be counted in surveillance. The person must seek care, have a stool sample submitted; the laboratory must test for the appropriate pathogen; and the laboratory test must be sensitive enough to identify that pathogen in the stool or other sample. Then it's reported to public health. If there's any break in the chain of these events, the illness is not reported or captured in surveillance.

Second, foods may be contaminated by many different etiologic agents -- bacteria, viruses, parasites, also chemicals and toxins. Some of these data have well-established surveillance systems. Others don't have any routinesurveillance data, and the example I've used here is norovirus. We consider this to be an important contributor to the burden of foodborne illness, yet there is no routine surveillance. Fortunately, now we have some information on outbreaks, which you'll hear about in a moment. But for the time period we're looking at, there was no routine surveillance.

Third, it's likely that an important proportion of foodborne illnesses are due to unknown agents, or agents that we know about but are not recognized as causing foodborne illness. This doesn't seem to be such an out-of-the-box idea, since many of the foodborne pathogens that we're most concerned about now, like E. coli O157, were only discovered in recent decades, E. coli O157, in 1982.

And finally, as has been previously mentioned, the things that we consider to be foodborne pathogens can also be transmitted by water or by animal contact or by other transmission routes. So in order to estimate the burden of foodborne illness, we really have to have a good understanding of the epidemiology of that pathogen, and be able to determine or estimate what proportion is actually attributable to food.

So I think many of you are aware of the estimates published by CDC in '99. Paul Mead was the lead author on that paper. And this was really the first attempt to comprehensively, across a broad array of pathogens, including unknown agents, to estimate the total burden of foodborne illness in the United States.

In the intervening 10 years, in FoodNet and in other groups at CDC, we have systematically tried to address some of the data gaps and some of the methodological issues that were identified in that paper in '99. So we're currently revising these estimates. We're using new, refined methods, learning from other studies that have been conducted in Europe and Australia, and also within the United States, in the intervening 10 years. We're using more recent data and new data sources where they are available.

We've also chosen to focus on domestically acquired foodborne illnesses. And this is a direct response to calls from regulatory partners who want to know what is the burden of illness caused by foods that are actually consumed in the U.S., and not associated with foreign travel.

So the aim of our study was, therefore, to estimate the burden of domestically acquired foodborne illnesses, hospitalizations, and deaths in the United States. Specifically, we wanted to estimate the burden due to 31 pathogens that are commonly transmitted through food, and then unspecified agents. And this category to us included unknown agents, agents that are known but not recognized as causing foodborne illness, and some chemical foodborne agents where we really don't have the data necessary to estimate their burden separately.

So the one thing I wanted to mention up front, because I certainly don't have enough time to go into this in detail throughout my presentation, is one of the major methodological advancements I think we've made to the estimates published in '99. And as you'll see in the next few slides as I go through the methods, these estimates are made up of many, many, many, many inputs. Each input carries some uncertainty with it. To reflect this uncertainty we've therefore used probability distributions to describe each and every data input, using empirical or Beta/PERT distributions and reflecting the uncertainty in this analysis.

As a result of this uncertainty analysis, our model outputs are in the form of probability distributions. So we don't have this as the number, but rather describe a best estimate and then a 90 percent credibility interval around that estimate.

That's all I'm going to say about the model, but just to be aware as I go through the slides, when I talk about inputs, I may talk about them as our best estimate. But know that every single one of them has a probability distribution surrounding them.

So as I go through the methods, I've divided it into kind of two broad sections. First, I'm going to talk about how we estimate the burden due to the 31 pathogens commonly transmitted through food. This is kind of subsetted into two groups: the pathogens for which we have surveillance data, and the pathogens for which no routine surveillance data are available. Then I'll talk about the unspecified agents.

Because of the limited time we have here today, I've chosen to focus on how we estimated the illnesses and will not go into detail on the hospitalizations or deaths, but I'd be happy to discuss that with anybody who's interested during the breaks.

So these are the 31 pathogens commonly transmitted through food that we've included in our assessment. It's a pretty familiar list when compared to the '99 assessment. There are a couple of additions. We've added Mycobacterium bovis and also sapovirus, which was only recently identified as a foodborne agent. And the Vibrio groupings – we separated out Vibrio parahaemolyticus, which was previously included under “Vibrio other.”

The pathogens which are highlighted in white are the pathogens for which we had surveillance data, so they total 25. And for the remaining six pathogens, we have no surveillance data available.

So let's first talk about how we estimate the burden for those pathogens for which we have surveillance data. But for each of these pathogens, we know, at the top of our surveillance pyramid, how many cases are being reported to foodborne disease surveillance. What we want to estimate is the bottom of the pyramid, or some fraction of the bottom of the pyramid, what fraction of those are foodborne and domestically acquired in the community. So to do this, for each and every one of those 25 pathogens, we adjust for underreporting and for under-ascertainment, and then estimate what fraction are domestically acquired and foodborne. And I'm just going to quickly take you through each of those steps and how those steps are implemented.

So our first goal is to try and estimate the number of illnesses due to these 25 individual pathogens that are reported to surveillance. So we use a number of surveillance systems, and they're listed here under three categories of active, passive, and outbreak surveillance. You'll be familiar with FoodNet. We also have data from the National Notifiable Disease Surveillance System, the Cholera and Other Vibrio Illnesses Surveillance System, and for M. bovis, we use data from the National Tuberculosis Surveillance System. And then finally, outbreak data from the then Electronic Foodborne Outbreak Reporting System, or eFORS.

When data were available from more than one system, we went with FoodNet. And we only used outbreak data when there were no individual surveillance data available.

So now we begin our journey. We have the top part of our pyramid, which is our estimate of the number of reported illnesses. Once we determine the number of reported illnesses, the next step is to adjust for underreporting. Because FoodNet conducts active surveillance, we assume that all laboratory-confirmed cases were enumerated by FoodNet. So FoodNet was our gold standard. If you had Salmonella or Campylobacter in a FoodNet site that was lab-confirmed, you were going to be reported.

We know that's not the case for passive surveillance. In passive surveillance, you're relying on the physician or the laboratory to remember to report and to report. So there is likely to be some underreporting.

To adjust for underreporting, we derived an underreporting multiplier. We did this by comparing the estimated cases in FoodNet to the cases reported to National Notifiable Disease Surveillance System for a common set of -- for a group of pathogens reported to both.

For outbreak surveillance, only outbreak cases are going to be reported, so we had to estimate the total number of outbreak cases that would have been -- the total number of cases that would have been reported, and adjust for underreporting. To do this, we again looked at FoodNet data and looked at what proportion of those cases were outbreak-associated, and derived a multiplier based on that information.

So after adjusting for underreporting, we have a good estimate of how many illnesses were identified in the laboratory for each of the 25 different pathogens. The problem is, we know that laboratories don't routinely test for all of these 31 - or 25 – foodborne pathogens that we're interested in. Also, laboratory tests, we know, are not going to be 100 percent sensitive. So once we’ve estimated the number of illnesses identified in the lab, we need to adjust for under-diagnosis because of test sensitivity or laboratory testing practices. So data on laboratory testing practices and laboratory tests were available, in many cases from FoodNet, which conducts periodic laboratory surveys asking about practices and the number of stools submitted and tested and positive. And for those pathogens not included in FoodNet, we turned to other published studies and created a multiplier for each pathogen. So after adjusting for lab-test sensitivity and laboratory testing practices, we now have a good estimate of the number of specimens that were submitted to that laboratory. Next, we need to adjust for medical care-seeking and stool-sample submission to estimate the total number of illnesses in the community. We did this using data from FoodNet population surveys. These are telephone surveys of the general FoodNet population, and they collect information on diarrhea and vomiting and on medical care-seeking behaviors of those persons who are ill.

Now, obviously, not all persons are going to seek care and submit a stool sample at the same frequency. If you're more ill, you're more likely to seek care. So we use bloody diarrhea as a proxy to adjust separately for persons with bloody or non-bloody diarrhea, and we did this for each pathogen. For example, from the FoodNet population survey, we know that 43 percent of those with bloody diarrhea seek care, and 39 percent will submit a stool sample. This compares to just 19 percent of those with non-bloody diarrhea, and 16 percent of those submit a stool sample.

So for example, for E. coli O157, which has a higher rate of bloody diarrhea, more of those persons would have been adjusted at the rate of 43 and 39 percent, compared to something like Campylobacter, which would have a lower percent bloody diarrhea.

So just to recap, we've started with the number of reported illnesses; adjusted for underreporting, test sensitivity in laboratory testing practices, medical care seeking, and stool sample submission; and we've done this for each of the 25 pathogens, to estimate the number of persons ill in the community.

Next, we need to estimate what proportion of these are domestically acquired and what proportion are foodborne. And we did this using a range of data sources, including surveillance, risk factor studies, outbreak data, and the published literature.

So next I want to turn to how we estimated illnesses for pathogens with no surveillance data available. We use a very different approach for toxoplasmosis and for some of the viruses than we did for bacteria. And I think this is important, because it represents one of the data gaps that we'll hope to close in the next iteration of these estimates.

As we've just gone through for those pathogens where we had surveillance data, we started with the top of the pyramid and adjusted downwards to estimate the bottom of the pyramid or some fraction of the number of cases that were domestically acquired and foodborne. Because of a lack of surveillance data for toxoplasmosis and for some viruses, we had to start at the bottom with some estimate of those at risk or the number of illnesses, and then estimate upwards what fraction of those were domestically acquired and foodborne. Just quickly to go through the methods that we used for toxoplasmosis, for example, we used zero prevalence data from NHANES, the National Health and Nutrition Examination Survey. In an estimate of what proportion of those seroconvert – in what proportion of them seroconversion is associated with clinical illness.

For astrovirus, rotavirus, and sapovirus, it was assumed that 75 percent of children experience a clinical episode of each of these illnesses by the time they're five years of age. And we applied this proportion to the 2006 U.S. Census birth cohort. For norovirus, our method is similar to that used previously. Because there’s no surveillance data, we have to rely on data from other developed countries, where it's estimated that 11 percent of acute gastroenteritis is due to norovirus. We took this percentage or fraction and applied it to our estimate of acute gastroenteritis from the FoodNet population surveys.

That was a very brief overview of our 31 pathogens. I want to shift gears now and talk about how we estimated the burden due to unspecified agents; again, unknown agents, those that are unrecognized as causing foodborne illness, and the chemical and toxins for which we were lacking data.

So to estimate what we don't know, we started with a syndrome, with acute gastroenteritis. I mentioned the FoodNet population survey, which asks the general population about symptoms of diarrhea and vomiting. Well, from that, we were able to determine the burden of acute gastroenteritis.

So we know some fraction of these illnesses are going to be due to the gastrointestinal pathogens for which we've already estimated the burden of illness. So by subtracting these, we're left with our estimate of the acute gastrointestinal illnesses due to unspecified agents. We know that some fraction of these are going to be domestically acquired, and some fraction are going to be acquired while traveling outside the United States.

Unfortunately, because we're estimating what we don't know, we don't know how many of them were acquired internationally. So we have to turn to the pathogens that we do know about. So for the 25 gastrointestinal pathogens, we looked at what fraction of those were then travel-related, and excluded that proportion from our estimate of unknown gastrointestinal illnesses.

So we're now left with an estimate of domestically acquired gastrointestinal illnesses, some fraction of which are foodborne and some fraction of which are transmitted by other routes. Again, we don't know what that proportion is, but we can turn to the 25 gastrointestinal pathogens that we do know about and apply that foodborne fraction to these unspecified agents. And that's how we end up with an estimate of domestically acquired foodborne gastrointestinal illnesses due to unspecified agents.

So that was, as I said, a bird's-eye view of the methods that we've used to estimate the burden of disease. A question that comes to mind is, then, how do these estimates compare with those previously published in '99? And I'm sure for many who are familiar with the '99 estimates, even with this bird's - eye view, you can see that there are important differences between those estimates and these.

These estimates have generally followed that previous approach, but have extended and enhanced many of the methods used, using new or more recent data whenever available.

Because of these changes, it's very difficult to compare directly the estimates in '99 to the current estimates. In a way, it's like comparing apples and oranges. We've changed many of the methods, and we've changed many of the data inputs, which is fine, because we have FoodNet surveillance data to look to for trends.

In the last few slides, I just wanted to highlight what some of those main methodological and data source changes were. I think the methods fall into three broad categories. The first is the exclusion of travel-related illnesses. So in the previous estimate, we had an estimate of all foodborne illness, whether it was acquired abroad or not. In response to calls from our regulatory partners, we have come up with an estimate of foodborne illness caused by food consumed in the United States, which I think is an important enhancement to these estimates.

Secondly, we applied pathogen-specific multipliers, increasing the precision of our burden of illness estimates for each of these 25 pathogens. And what I mean by this is that the process I just brought you through, starting with the number of reported illnesses, down to the total number of illnesses in the community, was assessed for each pathogen individually, rather than grouping pathogens together and applying a kind of a blanket multiplier, which was done previously.

And finally, I can't underline enough the importance of including this uncertainty model to describe the range of uncertainty that we have around the estimate overall and individual pathogens. For some, we're more confident with our estimate than for others, and that allows that to be highlighted. There are also examples of new data from the FoodNet population survey, new data on the norovirus faction. And –there are numerous other changes, which I'll give you a couple of examples of.

We have completed five iterations of the FoodNet population survey. The '99 estimates used data from the first FoodNet population survey, which collected information on diarrheal illness and then had to look to earlier U.S. studies for information on vomiting illnesses. The current study uses data from more recent FoodNet surveys, which collect both components that were needed to estimate acute gastrointestinal illnesses. The result of this is that we have a lower estimate of acute gastrointestinal illnesses, 0.6 compared to 0.79 in 1999. So this lower estimate is a much lower estimate than in the earlier study, but is really due to changes in methodology rather than significant changes in the overall rate of illness.

The FoodNet population survey also comes into play when we talk about our under-diagnosis multiplier. If you remember, we're taking the number of reported illnesses and we're adjusting for underreporting. And one of the key components of this is how many people are actually seeking care. Well, we have, again, more precise data from the FoodNet population survey. The '99 estimate, which relied on the first survey, which included just 9,000 interviews, estimated that 12 percent of people seek care. All of the other subsequent surveys estimate that rate to be 20 percent. We believe this is a more precise and more accurate estimate of care-seeking. But it resulted in a smaller multiplier, which resulted in a lower number of illnesses, regardless of any trend.

Another new data source which has an important impact is data on the proportion or fraction of norovirus illnesses that were considered to be foodborne. We have more precise data now on the proportion that are foodborne. More recent data on norovirus outbreaks estimate that 27 percent of illnesses are foodborne. This compares with 40 percent in '99. This results in not only a lower estimate of the number of norovirus illnesses that are foodborne, but because norovirus contributes to the overall estimate of all gastrointestinal illnesses that are foodborne, it results in a lower estimate of foodborne for those unspecified agents.

There are many other new data sources, and I certainly don't have time to go through them all today. But just wanted to give you one example, and that is for Yersinia. In '99 that was assumed to be similar to Salmonella with regard to the under-diagnosis multiplier. So Salmonella, there were 38 for every one in the community, and the same was considered for Yersinia. In the current analysis, we didn't rely or assume that Yersinia was like Salmonella. Rather, we went through the pyramid and identified data specific to Yersinia. We came up with new data on the proportion severe, new data on the frequency of laboratory testing for Yersinia, and new data on test sensitivity for Yersinia. The new Yersinia-specific multiplier is very different to that for Salmonella, and this is carried through for the other pathogens for which we calculated this precise individual pathogen multiplier.

So in conclusion, these will provide new estimates of the burden due to domestically acquired foodborne illness. Because of methodological changes, these estimates cannot be directly compared to those published in '99 for the purposes of assessing trends. Data and methods gaps definitely remain, and I think one of the purposes of this paper will be to identify what those are and to help set the agenda for what we should work on over the next five to 10 years before we repeat these estimates.

Regardless, these estimates, along with the other metrics we have discussed today, will hopefully help direct food policy and interventions. Thank you. (Applause)

DR. BRADEN: Thank you, Dr. Scallan. I'd like to next introduce Dr. Barbara Mahon. She'll be speaking on the topic of tracking foodborne disease outbreaks due to specific food commodities. Dr. Mahon is a leader of the FoodNet and Outbreak Surveillance Team of the CDC's Enteric Diseases Epidemiology Branch. Dr. Mahon's team conducts surveillance and related epidemiologic studies to improve knowledge about the burden, trends, attribution, and risk factors for foodborne illnesses.

Dr. Mahon is a pediatrician who has taught at several medical schools. She has a faculty appointment in epidemiology at the Boston University School of Public Health. Please welcome Dr. Mahon. (Applause)

DR. MAHON: Good morning. I'm going to be talking about CDC's outbreak surveillance system and how we use that system to track illnesses due to specific food commodities.

So just to start at the beginning – the definition of a foodborne disease outbreak. We consider a foodborne disease outbreak to be an incident in which two or more persons experience a similar illness resulting from ingestion of a common food. Now, outbreaks in the United States are almost always investigated and reported by state and local health departments. CDC receives voluntary reports of the results of those investigations from the reporting agencies. The CDC itself is involved in foodborne outbreak investigations only in a very small minority of cases when either it's a multistate outbreak, in which CDC helps with coordination of the outbreak investigation, or by invitation of the state and local health department, if the outbreak is particularly large or particularly difficult. So in the vast majority of cases, the investigation is done by the state or local or territorial or tribal health department and then voluntarily reported to CDC.

We accept reports of outbreaks due to really any cause -- bacterial, viral, parasitic, toxic, and so forth, whether intentional or unintentional. And the data that are collected are quite extensive. I've just summarized them in this slide. The actual number of data fields is pretty long. So what's included in the report is information about the location, both the state where the outbreak occurred and the setting – a school, a restaurant, a private home, and so forth; the magnitude of the outbreak, in terms of the number of illnesses, number of hospitalizations, number of deaths; and then the demographic characteristics of the people who were involved in the outbreak.

We also collect information, of course, on the pathogenic agent, the bacteria, virus, et cetera, that caused the outbreak, and on the implicated food vehicle, when a food vehicle is identified in the course of the outbreak investigation.

We are able to collect information on contributing factors, those factors that led to contamination of the food vehicle, when this information is available. Very often it's not available, but we do collect it when it is available. And we also collect information on the public health response.

This slide shows the number of outbreaks reported to CDC by year, from 1973 through 2008, in the colored bars. The outbreak surveillance system was enhanced in 1998, which explains the large increase that you see right around that time period. That's due to an increase in surveillance activities, not to an increase in outbreaks, per se. And you'll see in the white line the number of outbreak-associated illnesses reported over that time period.

States vary substantially in the number of foodborne disease outbreaks that they investigate and report. This map shows for the five-year time span from 2003 to 2007 the number of outbreaks per 100,000 population per year reported by different states. And it's divided into fifths, from the highest fifth to the lowest fifth. And what you'll see is that the states that show up dark green in this map are the states that are in the highest fifth of reporting. These states reported an average of one outbreak per 100,000 population per year during this time period. The states that are in the lowest fifth had only a twelfth of that number of outbreaks.

Now, this doesn't mean that, in the light green states, that food is 12 times as safe in those states. It does mean that the dark green states are doing a lot more outbreak investigation and reporting.

One of the key elements of an outbreak report is, of course, the pathogen etiology; what was the bacterium or virus that caused the outbreak. And this slide shows the reporting of pathogenic etiology from 1998 to 2008. What you see is, in the bright green, the proportion of outbreaks with confirmed etiology that has gradually increased over this time period, and currently is at about 50 percent of outbreaks, more or less. Over the same time period, the number of outbreaks with no etiology reported, which are in gray, has decreased and currently stands at about 30 percent of outbreaks. Many of those with an undetermined etiology are the very small outbreaks for which an investigation is often less likely to determine etiology. The purple bars are those with a suspected etiology.

Now, another important component of the outbreak report is the implicated food. And before I show you information about implicated foods, I just want to tell you how we characterize foods. More than 1,800 different foods have been reported to CDC's outbreak surveillance system. So it's necessary to have some sort of categorization to be able to add up the outbreaks and the illnesses that are attributable to any particular kind of food.

So this slide shows a categorization scheme that we're using, in which all foods, at the top, are divided into three major categories: land animals, plants, and sea animals. And then each of these categories is further subdivided to yield categories that are more useful for analysis. So, for example, for land animals, you can see that the meat and poultry category is subdivided into beef, pork, poultry, and game.

For land animals and sea animals, the categorization was fairly simple. For plants, it's more complicated, because there are so many different kinds of foods that come from plants. So if you imagine, if you envision going to the grocery store, you have the produce section, and the produce section includes both fruits and vegetables. This is divided into the fruits/nuts category, and then the vegetables are divided into several categories: leafy greens, root vegetables, vine/stalk vegetables, sprouts, and fungus. And then we have the grains and beans, which are a category of their own. And then there's a group of other plant products that we summarize as oils and sugars that are of plant origin and are processed and form their own category. So this slide shows the proportion of outbreaks over that same time period for which an implicated food was reported. And what you see is that it's approximately 50 percent where an implicated food is reported, and that, again, the smaller outbreaks are less likely to have an implicated food reported.

Now, we need to take another step with understanding these implicated foods. So not all foods will fall simply into one of the commodity categories that I talked to you about in the previous slide. Some of these foods are simple foods, what we call simple foods, where all the ingredients of the food come from one commodity group. So, for example, steak is beef. That's easy. It maps to the beef commodity, and it's clear-cut where it belongs. Fruit salad is also considered a simple food. Although fruit salad can be made up of many different kinds of fruits, all of those come from that fruits and nuts category. So we consider that a simple food, and the illnesses that occur in an outbreak that's due to a simple food can all be attributed to that commodity category.

There are many implicated foods, however, that are not simple. We call them complex foods. And these are foods that are made of ingredients that come from multiple commodity categories. Meatloaf is a good example. Meatloaf typically is made from beef, from eggs, from -- how do you make your meatloaf? (Laughter.) From bread, from tomato sauce. And so those are going to map to a number of different commodity categories. The ground beef would map to beef, the eggs would map to eggs, the bread would map to the grains and beans category, and so forth.

Now, these -- attributing the illnesses that are due to outbreaks caused by complex foods is quite a bit more complicated than saying what commodity the illnesses and outbreaks due to simple foods are due to. Dr. Griffin, in her talk later in this session, is going to talk about our method for doing that. This is a method that's in development. What I'm going to show you is based on analysis of simple foods alone.

So when we now look at the information on implicated food status that I showed you a couple of slides ago, you can see that the simple foods, which are in green, have typically made up something like 20 percent of the outbreaks that have been reported to CDC across the years. And when we look at the food commodities that have been responsible for those outbreaks, we see that they fall out something like this.

This slide captures the illnesses in 1,355 outbreaks that were reported to CDC from 2003 to 2007 that were caused by simple foods; those that include only one of the commodities that I mentioned to you. And we'll see that about 21 percent were attributed to poultry and about 14 percent to leafy greens, and then about 10 percent each to beef, to dairy, to fruits/nuts, to vine/stalk vegetables like tomatoes, and so forth.

This sort of information is published on an annual basis. This is a front page of an article that was published in the MMWR, last June, that summarizes a year of outbreak reports, including the simple commodity attribution that I've just discussed. And in the course of doing this analysis, we can report the number of illnesses, hospitalizations, and deaths, overall, and then due to outbreaks caused by specific pathogens, due to outbreaks attributed to specific commodities, and then to commodity/pathogen pairs. I wanted to mention that we have recently made available a new online tool. It's available at the URL that's listed here. It's called the Foodborne Outbreak Online Database, and this is a publicly available search tool that anyone can use to search by state, by year, by setting, by pathogen, for outbreaks that have been reported to CDC. And you can then download information that includes the number of illnesses, hospitalizations, deaths, and the implicated food, when a food was implicated. `There are a number of limitations of using outbreak data to attribute illnesses to foods, and many of them are fully apparent from the information that I've already shared with you. Most foodborne illnesses are not part of outbreaks. The vast majority of illnesses occur singly, so they're not part of recognized outbreaks. However, even when they are part of outbreaks, we have incomplete information that's available to us for analysis. So we know that large outbreaks with short incubation periods are most likely to be evident, to be investigated. And smaller outbreaks, or outbreaks with shorter incubation periods, are therefore less likely to be investigated in the first place.

Even when outbreaks are investigated, there's incomplete reporting to CDC. And when outbreaks are reported, there may be incomplete information. A full investigation can often be done and not yield all the information that we ideally would like to have about the outbreak, so our information is incomplete.

Also, there's variation in outbreak investigation and reporting over time. And this variation makes the interpretation of any trends that we may see in the outbreak very difficult to do. It's hard to know what to attribute to simple variation in outbreak investigation and surveillance, as opposed to any real trend that may be occurring.

Finally, the commodity categories that I've mentioned, like any sort of categorization, does include a degree of inherent arbitrariness. They make some sense, but they're not perfect for all situations. So, in particular, it may well be that specific subgroups within some of these categories are responsible for a disproportionate number of the outbreaks that are attributed to that category. For example, unpasteurized milk is clearly the part of the dairy category that's most concerning, causing the most outbreaks and the most illnesses. But in the current commodity scheme that I've showed you, it's combined with all dairy products. Some of our plans for improving on the outbreak surveillance that we're currently doing are listed here. We are going to continue with our annual reports of outbreaks, together with the simple commodity attribution that I've presented to you, and publishing that each year in the year after data close- out.

We also are working on refinements and enhancements to the food commodity classification that will allow increased specificity and allow increased flexibility in responding to specific questions that may come up about the details of those commodities. And then, finally, we're developing and implementing methods to use outbreak data for more complex attribution analyses. And, again, Dr. Griffin is going to be presenting some of this work later in tis session.

So I'd like to acknowledge the many people who have contributed to this work, and I will close it ere. Thank you.(Applause) DR. BRADEN: Thank you, Barbara. I'd now like to introduce our next speaker, Dr. Tauxe. He will be presenting an overview of methods for attributing the burden of illness across modes of transmission and food vehicles. Dr. Tauxe has a long history in food safety. He is currently the Deputy Director of the CDC's Division of Foodborne Bacterial and Mycotic Diseases, which is the division charged with prevention and control of intestinal bacterial, zoonotic, and fungal infections.

The division monitors the frequency of these infections in the United States, investigates the outbreaks, and develops strategies to reduce the disease, disability, and deaths they cause. Dr. Tauxe's interests include bacterial enteric diseases, epidemiology and pathogenesis of infectious diseases, epidemiologic and clinical consequences of bacterial genetic exchange, antimicrobial use and resistance to antimicrobial agents, and teaching epidemiologic methods. He has served internationally and has supervised numerous overseas epidemiologic investigations. Dr. Tauxe, who is certified in internal medicine, has faculty appointments including at the School of Public Health, Department of International Health, and the Department of Biology, at Emory University. Please welcome Dr. Tauxe. Applause)

DR. TAUXE: Thank you, Dr. Braden, and thanks to all of you. It's an honor and a pleasure to be here this morning. I'm going to summarize some thoughts about an overview of our efforts, at CDC, to address this issue of food attribution from a number of different angles. And I'm going to present a framework that I, at least, find helpful in describing attribution efforts.

Attribution is a word that is certainly in the air a great deal. We use it to mean -- when we're talking about the attribution of illness to food commodities -- we use it to mean allocating the burden of illness, either from a specific pathogen or from all pathogens, to different foods or to other routes of transmission. We're not using this term, at least we don't use this term, to describe the analyses within a single outbreak that link that illness and that outbreak to a specific food. For us, that's the implication process implicating a particular food.

CDC surveillance systems provide reliable data on the incidence of illness caused by a number of specific enteric pathogens. But we really don't have a system, and there is no system currently operating, to determine the illnesses that are caused by consumption of a range of different foods.

Illnesses can be attributed to foods using a range of data sources and analytic methods, and as we do this, we can consider several different dimensions. I'm going to start with those dimensions, to frame how we think about these things. One dimension is the pathogen dimension. There is a whole array of different pathogens that can cause foodborne diseases. There are the infectious pathogens -- the bacteria, the viruses, the parasites. There's the prion, whose status alive or dead we can discuss separately, and the toxins and so forth. There is a range of different pathogens or agents.

Second, there is the vehicle dimension. And this was just outlined as animal foods, produce, seafood. But in fact, there are other vehicles that these same organisms or pathogens can transmit through -- the water that we drink or swim in or bathe in; pets; people-to-people transmission. There are other routes of transmission, as well, and we need to bound our discussions when we're talking about this.

And if we do that, and here is the bounded plane of pathogen and vehicle, speaking just about the infectious diseases now, and leaving out chemicals/toxins for the moment, we have -- we can think of a problem being in one or another part of this plane as related to a particular food or a particular pathogen or the whole plane.

There is, of course, a third dimension that's already been mentioned extensively this morning, and that is the farm-to-table continuum, or the food-processing continuum, that starts with whatever it is that animals or plants are fed with or watered with, and includes production at the production level – the farm, orchard, or fishery; the food passing through processing at the slaughter plant, the packer, the cannery. Finally, the preparation in the kitchen or the food-service establishment. And, you know, you can think of that as continuing. Okay, then here is the table, and then unfortunately, sometimes there's pathology that follows. After a period of incubation, there is illness or death. And if we want to be discussing how we do attribution, then we need to sort of think of the range of what's inside bounds and what's out of bounds. And I think of this as the food safety box, which is the range of foods we're talking about, the range of infectious pathogens we're talking about, and the range of the farm-to-table continuum we're talking about.

There are a number of different methods that we can use to approach how to sort out the attribution, and, very briefly, three main categories. One is expert opinion, surveying expert opinion. Say, using the Delphi method or other methods, and asking a group of experts to tell us what do they think.

The second main method is epidemiological approaches. That includes the analysis of outbreak data that we were just hearing about a bit, and case/control studies of the sporadic illnesses. And then there are microbiological approaches that depend on the isolation data, collections of pathogens from foods or animals, and comparing that with what comes from people. And this is much strengthened when subtyping data are available. Each of these has strengths and limitations, and no one provides the final answer.

To mention very briefly the Delphi method, his is something that we certainly participate in at CDC. And we've used expert opinion to fill in a few gaps when there are simply no other data available, but it's not a primary tool for us. It's named for the Oracle at Delphi on Mt. Parnassus in Greece, and this is a Victorian-era image of the Oracle becoming intoxicated by the volcanic fumes coming out of the crypt which was in the cave where she was. And she would deliver oracular statements which would be interpreted by others, and was a source of inspired truth in the classic age.

We don't depend on volcanic emissions, but for us, it's a structured solicitation of expert opinion where there is inherent uncertainty. A poll is conducted, of a panel, by questionnaire, fed back; they get the results, and they are re-polled, and a systematic collection of this subjective, collective judgment that's used for a number of purposes where quantitative assessment is not possible. And I have a great deal of respect for this method, and it's underway in several parts of the food safety arena.

Now, we use other methods; epidemiological methods, for example, one of which is the epidemiological case/control study, which is our best way of determining a number of things about sporadic cases -- cases of illness that are not part of recognized outbreaks. And I'm not going to go through, in detail, how to do an epidemiological case/control sudy, but at its heart, it means defining cases of sporadic infection, usually with a surveillance system, and FoodNet is really a platform that allows important sporadic case/control studies to be done that otherwise could not happen. It defines the cases, and then we interview them systematically about a range of exposures, and then we find healthy control people and interview them about the same range of important exposures. And using that, we conduct a statistical comparison of the frequency of reported exposures or underlying conditions or other risk factors, and can calculate statistical measures of association and, in some sense, can measure the fraction of the illnesses that may be attributed to each of those exposures.

With a case/control study, we can consider a whole range of different exposures at once, including food and non-food exposures. We do have to focus on a single pathogen, because that's the cases that are being defined and considered.

So FoodNet has conducted quite a number of case/control studies at this point, which I've listed here in print that's probably not legible from the back of the room. But just for -- to skip through, for example, for Campylobacter, the primary risk factors identified were consuming undercooked poultry or other meats, animal contact, foreign travel. So a case control study like this can pick up the non-food exposures and estimate proportions that might be related to that.

The Salmonella Enteritidis case control studies, several of them over time, clearly show the risk associated with eating undercooked eggs, but the growing importance of undercooked poultry, as well. And one recent case control study just published last year, I believe, was of Toxoplasma . This is actually not a FoodNet study, but it was conducted by CDC. Handling or eating undercooked meat and having kittens in the home were the risk factors, and the undercooked meat was -- the contact with undercooked meat was much more important than the kittens.

Now, these case/control studies have been done and contribute very useful information about sporadic cases, but they do have limitations. First of all, the sporadic cases themselves have to be diagnosed, and the ones that are diagnosed are at the more severe end of the spectrum, probably. Our whole process depends on human memory, what people can report about their exposures, which may be, as we've heard, three to four or more weeks afterwards.

And the only exposures that can be surveyed or commented on or studied are the ones that the person or their family would be aware of. Exposures that they really can't identify really are hard to study in this way. And exposures that are very common in the general population are difficult to identify as specific risk factors. If everybody is doing something, it's hard to find a difference between ill and well. It does depend on the general population being non-susceptible. If almost everybody were immune to an infection, then it wouldn't matter what they did, and we wouldn't find differences.

It does depend on a high rate of exposure to the suspect vehicle, so we can only identify the strongest signals, not associations that only account for a small number of cases. And if there's a low -- and we assume there's a relatively low rate of contamination in the food. And we think of this information as providing signals of risk, rather than a full, precise quantitation.

Now, a second method that's already been addressed in some detail by Dr. Mahon is using outbreak data. And I'll just comment here to say -- to think, well, all right, if we have case/control studies on a number of specific pathogens, how about the outbreak data? Why would we use outbreak data to allocate illness by food commodity? We do get, as Dr. Mahon noted, over 1,200 foodborne outbreaks reported each year. And for many pathogens, this is the most conclusive indication we have of where illnesses -- which foods caused the illness. We don't have case/control studies for a large number of pathogens. And this gives us information about a much larger range of pathogens. It represents, to varying degrees, all the foodborne pathogens, many of which are not included in other surveillance systems. It also represents and captures a very wide range of food vehicles, since we just collect the information about what was implicated, and don't approach it with a preconceived list of things that might be risk factors. And so it can identify vehicles that we might have overlooked in the past in other sorts of surveys, like sprouts or almonds or raw cookie dough. And it captures the effect of contamination that occurs at all points, from the farm to the table.

Challenges involved in using the outbreak data, of course, depend -- first of all, this is dependent on the capacity, skill, and resources available at State and local health departments. And if they don't investigate, then we don't get reports.

It also depends on the challenge that was just outlined, that many outbreak food vehicles have, in fact, multiple ingredients. There is also the challenge of integrating the information or blending the information that we have from outbreaks when there is a case/control study also available; how do we -- and this is a challenge we have not successfully met yet -- how do we fully integrate the information from the two sorts of studies about the same pathogen?

And, finally, there are some pathogens that rarely or never cause recognized outbreaks, though we do think they are foodborne. And so we don't really get at them this way.

Now, I mentioned that it captured the universe of foods, and the universe has several globular clusters or galaxies in it. And we break those up. This is just as described in the food categorization into the simple foods and the complex foods, and I think, as a society, we tend to really be very fond of complex foods. An awful lot of what we eat is complex in this sense. And we categorize those into those 17 commodities that were just outlined.

Now, we can, as Dr. Mahon described, limit our analysis to data from outbreaks with simple foods, say, including outbreaks due to steak, but not meatloaf, in our analysis as a way of attributing illness to beef. Or we can include data from outbreaks with both simple and complex foods, determining the ingredients of the complex foods and modeling the relative importance of each ingredient, as we will hear shortly, I think, from Dr. Griffin.

As we do this, I think we need to be remembering ourselves, now, what is it that we actually are seeing or saying or talking about? What level of attribution are we operating at? And for all of these methods that depend on the patient interview, the information about what the person ate, we're talking about attribution at the point of consumption, at the point at which the food was actually consumed. Which is, of course, at the bottom of the farm-to-table continuum. And so whether it's case control/studies, asking people what they ate before they got ill, or if it's summaries of outbreaks depending on implicated vehicle reports, we're talking about this plane and various ways of estimating what's going on at this level of the plane – point-of-consumption attribution.

Now, that's not the same thing as saying, gee, all that illness is beef or produce or seafood, down here; meat or produce or seafood, down here, goes back necessarily in a linear way to the farm, the field, or the harvest site, because many things can happen along the way. So we can think -- it would be nice to have point-of-processing attribution, which was at the -- talking about the source of illness or about the contribution of illness from each of the production points or each of the processing categories for those main types of food. This is much harder, because if we're talking to ill people, they have no clue where their food was processed, or where it might have been contaminated. In fact, it's difficult to do. It's very difficult to say what -- where food became contaminated in general.

But one approach to this that has been pioneered in Denmark is the comparison of microbial libraries -- now, this is just for Salmonella – for point-of-processing attribution. And Denmark now, each year, publishes their Salmonella account. This requires many Salmonella isolates from many foods, which are then subtyped -- DNA fingerprinting or other methods of subtyping -- and then those subtypes are compared between the foods; the distribution of those subtypes are compared for foods and people, and an allocation is calculated on that basis.

CDC and FSIS are collaborating on this using Salmonella collected in slaughter plants now, and it's a complicated method, and it's an interesting one. But it's limited to those microbes for which large libraries exist, and it's limited really to the foods which have been carefully sampled. And so that means it's not universally applicable, and if we wanted to sort of place it on this food safety box, that point-of-processing is thus one pathogen, Salmonella. It's for -- in this case, in our case, a -- whole range of animal-derived foods, of meats, and just one pathogen. So I represented that by this red bar here. It's not a full-scale attribution at all of all foods at that point.

So I'm going to leave you with this, saying, as we talk about attribution, it must be clearly defined what we're saying and what we're describing. Attribution can address different pathogens, different levels in the food safety system. CDC surveillance and study data, combined with other information, can be used for several types of attribution, and the outbreak data cover a broad spectrum of pathogens and foods, but it's all at the point of consumption. A combination of methods is what we need to have for the most complete picture. Thank you very much. Applause) DR. BRADEN: Thank you, Rob.

We will continue now with this grand tour of metrics having to do with data at CDC. Next we have Dr. Patricia Griffin, talking to us about the attribution of illnesses to food commodities, an approach using outbreak data, an introduction to which you've already heard.

Dr. Griffin is the Chief of the Enteric Diseases, Epidemiology Branch, at CDC. The branch does surveillance for cases of illness and for outbreaks. It does studies on human illness due to bacterial agents such as Salmonella and E. coli O157; tracks trends in illnesses, and analyzes data on the relationship of illnesses to particular foods. Branch programs include FoodNet, the National Outbreak Reporting System, the human arm of the National Antimicrobial Resistance Monitoring System, for enteric bacteria.

Dr. Griffin has supervised epidemiologic investigations throughout the United States and overseas. Her medical training includes internal medicine, gastroenterology, and mucosal immunology. She holds an appointment to the Emory University Rollins School of Public Health. Please welcome Dr. Griffin. (Applause) DR. GRIFFIN: Good afternoon.

I think a lot of this will start to sound familiar to you. I hope it does. So, why use outbreak data to attribute illness to various food commodities? For most illnesses, the causative food can only be determined if the person was part of an outbreak, and outbreaks capture information on both common and uncommon agents, and on both common and uncommon food vehicles.

So to look at outbreaks, we looked at a sample dataset of foodborne disease outbreaks from 1998 through 2004, to see what we could do about assigning illnesses to commodities. And so here in the first column, and I'm pointing on the right side of the room, the etiologic agent may be single, such as Salmonella, and was confirmed. The investigators are pretty sure it was Salmonella, but not positive. The etiology may be multiple – both Salmonella and norovirus were causing this outbreak. Or he etiology may be undetermined.

The next two columns talk about the food vehicle. It may be implicated. For example, we're sure that the outbreak was caused by beef or by lettuce. And usually when we say we're sure, I think a lot of the general public thinks that means that we isolated the organism from the product. It almost never means that. We use epidemiologic and statistical techniques to determine the food vehicle. And so many times, about three-fifths of the time, we can say it was implicated. Many times, though, it's not determined.

Well, does that mean it was a crummy investigation? I mean, it's an awful lot of these outbreaks in which the food vehicle wasn't determined. But as you heard earlier, if a small number of people get sick, like under 10, it's really, really difficult, with any technique, to be sure what the implicated vehicle was. And that can also happen when there's a big outbreak. For example, if people have a Thanksgiving dinner and everybody has turkey, dressing, cranberry sauce, mashed potatoes and peas, and two-thirds of the people get sick, it's really unlikely that our techniques can figure out the answer.

So these outbreaks turn out to be not very useful for assigning illnesses to commodities. We still find them very useful for other purposes. They tell us a lot about settings. They tell us a lot about etiologies of outbreaks. So we really want them to be reported, but they're not that useful for this particular purpose.

Similarly, these two rows, the multiple confirmed agents and the undetermined agents, are not that useful for assigning illnesses to particular commodities. So out of the eight -- at the very bottom right, you see 8,800 outbreaks during this time period -- you're left with the ones in green, the ones due to a single, confirmed or suspected agent and an implicated food vehicle. So these are the ones that we analyze to try to develop this technique of assigning illnesses to particular food vehicles.

So you've heard before that more than 1,700 food items have been responsible for outbreaks, and they were responsible in this particular dataset in those green boxes. And we could just write out a line list of 1,700 outbreaks and what all those different 1,700 foods are and say, here, these are the ones that make you sick. But what would you do with that much information? You need it sort of organized in a way that makes sense. And so for meaningful analysis, these items have to be grouped into broader categories of commodities.

And you've seen this hierarchical scheme before, so here it is. And I've really highlighted the ones in yellow. All of those are the end point. There's no further branch after the yellow box. And those yellow boxes are the 17 commodities into which we assign foods.

Now, many of you in this room would love to have some of those commodities subdivided, and sometime when we have a whole lot of people, we'll go back and subdivide all those commodities. But at this point, this is what we have. And you've also heard about food categorization. Simple foods contain single commodities -- steak, categorized as beef. Complex foods contain multiple commodities, for example, that meatloaf that was mentioned before contains ground beef, eggs, bread, and tomato sauce, and so meatloaf includes the four commodities to which those ingredients get assigned.

So here's that same slide I showed you before with the boxes in green. Those are the ones that are left over that we can use for this analysis. All the other outbreaks are interesting, they're important, they can be used for other types of analyses, but these are the ones we use for this analysis. So now I'm just going to focus on the ones in the green box, and I'm going to show you another table.

And this shows foodborne disease outbreaks due to a single, confirmed or suspected etiologic agent -- that's the one in the green boxes. And the first column breaks it down by whether the agent was bacterial, viral, parasitic, or chemical. And also it breaks down the food vehicles into whether the vehicle was simple or complex food. And at the bottom, you see that these outbreaks, and there were about 2,900 outbreaks, resulted in a lot of cases, about 180,000 illnesses. So of these outbreaks, an awful lot of them were due to complex foods. So we have about 1,500 due to simple foods, and 1,400 due to complex foods.

So what are we going to do with the meatloaf outbreak? So maybe we should just say it's meatloaf, so we'll say it's ground beef. So we'll assign it to beef, because I mean, meatloaf is mostly beef. But then somebody might say, wait a minute -- eggs are in that meatloaf, and if eggs weren't cooked, maybe it really was the eggs, even though a lot of that, the major component, is ground beef. And somebody else might say, well, maybe it's the tomato sauce. We've had some issues with tomatoes. Somebody else might say, well, if it's E. coli, let's say it's beef. And if it's Salmonella, let's say it's -- oh, I don't know. So what do you do? What would you want to do with these complex outbreaks?

So we sort of thought through that and came up with some -- so here's an option: just get rid of them. So then we're getting rid of half the outbreaks.

So here are the choices that we thought about. One is, limit the analysis to data from outbreaks with simple foods. So for example, we would include the outbreak due to steak, but not the meatloaf. We don't even have to deal with that meatloaf. But then we lose data from about half the outbreaks, and we lose a lot of data from foods that are typically consumed as part of complex foods, like lettuce. We typically consume lettuce along with tomatoes, so that's a leafy vegetable with a vine/stalk vegetable, and cucumbers, and we put other things in it as well. So we'd get rid of all the lettuce outbreaks, which some people might be happy about. But it wouldn't present as accurate a picture as we would like.

So choice two is to include data from outbreaks with both simple and complex foods. How would we do that? We could determine the ingredients of complex foods, and I haven't talked about how we do that, but if they don't tell us, we have ways of saying, these are what we think are the major ingredients of these complex foods; and then we model the relative importance of each ingredient. So we devised a method to model the relative importance of each commodity in outbreaks due to complex foods. So we make low, high, and most probable estimates for each commodity.

So let's say there's a complex food outbreak, such as the meatloaf outbreak. So in the low estimate, we assume that none of the illnesses were due to beef. None were due to tomatoes; none were due to bread; none were due to eggs. So it's as if we got rid of the complex food outbreaks. So that's the low estimate.

The high estimate is we would assume that all the illnesses were due to each of the commodities. So let's say there were 100 illnesses in the outbreak. We'd say all 100 were due to beef. All 100 were also due to tomatoes. All 100 were also due to eggs, et cetera. And we just put it in the model that way, to figure out the model. That would give us a high estimate for each commodity, because it's possible that each of those caused the outbreak.

Then there's the most probable estimate, which we think is the best method, and that's to partition the illnesses in each outbreak into commodities based on data from prior outbreaks, and assign illnesses only to commodities that have been shown to transmit this agent. So let's say it's a Salmonella outbreak. We'd say, if there hasn't been a Salmonella outbreak from bread in that dataset, we'd say none of the illnesses are due to the grains commodity. And then we'd look at other Salmonella outbreaks in that dataset that were simple outbreaks, but were due to beef or due to tomatoes. And looking at that data, assign those 100 illnesses, some to beef, some to tomatoes, some to eggs, according to prior data.

So then if we use this method, the next steps, which I've written down and then I'm going to show you schematically, so for each outbreak we determine the number of illnesses due to each food commodity. It's easy for the simple outbreaks, and then we use our little method for the complex outbreaks. And then for all outbreaks due to, for example, E. coli, we determine the percent of illnesses due to each food commodity. And we do the same thing for each agent. And then we take the burden estimates that Dr. Scallan told you about and obtain the number of U.S. illnesses due to each pathogen, for example E. coli.

And then we use the percents that we generated in that first bullet to determine the number of E. coli illnesses due to each commodity, and do this for all the agents and then add up all the number of illnesses due to each food commodity. So now I'm just going to show you this in table form.

So this is the hypothetical example. And you can see the percent of illnesses and outbreaks for E.coli. We can estimate in this example 50 percent of outbreak- associated illnesses are due to beef, none to pork; 40 percent to vegetables, and none to shellfish. Then we get that burden estimate that Dr. Scallan was talking about, the CDC estimate for the number of E. coli illnesses in the United States, and apply those percentages to that number.

And then we do the same thing for Vibrio, and there the percents will be very difficult, because outbreak-associated illnesses due to Vibrio are 95 percent due to shellfish. And we get that number of Vibrio illnesses in the United States, and apply the percent. And then we continue doing this for all the agents until we end up in the bottom right with the millions of foodborne illnesses due to known agents and the percent of those illnesses due to each food commodity.

And so when we do that, we can generate a graph like this, which is estimates of illnesses attributed to food commodities. Again, this is on a static dataset. It's preliminary data. It was just used to develop these methods. And along the X axis, you see those food commodities we've been talking about, and the blue square gives that most probable, that middle, that best estimate. The green gives the low estimate, and the red gives that sort of one where you assign every illness to every commodity in the complex outbreaks. So that's the sort of graph we can generate.

So there are a lot of limitations to this method. Outbreak data depend on reports from State and local health departments, and many outbreaks are not detected, not investigated, or not reported. And outbreaks that are large, associated with restaurants, that have short incubation periods, or cause serious illness are more likely to be investigated and reported.

The estimates are based on the numbers of cases occurring during outbreaks. And illnesses due to agents that cause many sporadic cases, but few outbreaks, may be underrepresented. And our method was devised using outbreak from '98 through 2004, but data from later years are now available.

And attribution of illness to a particular commodity does not indicate at what point in the commodity the commodity became contaminated, in that chart that Dr. Tauxe showed you. Illnesses also could be due to specific foods within a commodity. For example, ground beef in the beef commodity. So some of our plans are to use recent outbreak data to determine, for each agent, the percent of outbreak-associated illnesses due to each food commodity, and then when the improved burden of illness numbers are available, to estimate the percent of U.S. domestically acquired illnesses due to each food commodity, and to develop computer programs to help us in these complex analyses.

We want to continue the work each year of assigning illnesses in the over 1,000 outbreaks that get reported to us each year and assigning those illnesses to commodities; to create models to estimate trends in outbreak-associated illnesses due to particular commodities, and to improve foodborne outbreak investigation and reporting. In conclusion, outbreak data can provide estimates of the proportion of illnesses due to each food commodity, including all food commodities that have caused recent outbreaks, all agents that have caused recent outbreaks, and data from complex foods. The method relies on having the burden estimate of the number of illnesses due to each agent. And future possibilities include adding information from non-outbreak illnesses and measuring trends.

And, as in the previous work, many people contributed to development of this method. Thanks for your attention. (Applause)

DR. BRADEN: Thank you, Patty. For the last in this series, we have Dr.Cole. Dr. Cole will be presenting a method for estimating animal sources of human Salmonella infections, using microbiologic data from food product of animal origin. Dr. Cole is a veterinarian board-certified in internal medicine and a doctoral epidemiologist with the FoodNet and Outbreak Surveillance Teams in the Enteric Diseases Epidemiology Branch at CDC. Dr.Cole's primary activities include analyses of foodborne outbreak surveillance data and estimating the food source attribution of enteric diseases.

Previously, Dr. Cole worked as a clinical veterinarian in food and animal production medicine at the University of Georgia College of Veterinary Medicine, and as lead of the Vector-borne Zoonotic Diseases and Emergency Preparedness Epidemiology Teams at the Georgia Division of Public Health. She is currently an adjunct faculty member at the University of Georgia College of Public Health. Please welcome Dr. Cole. Applause)

DR. COLE: Thank you. And there's nothing like talking about meatloaf and steak all morning to make me really keenly aware that I am the last speaker before lunch, so I'll try to make this as painless as possible so that we can get on to hopefully have time for a couple of questions for our speakers and then get on to hopefully no foodborne illness, but at least the food part.

So I'm going to talk about a last method you've heard already somewhat mentioned, and I'm going to go into a little more detail about the methods that we use for estimating animal sources of human Salmonella infections. So first, a brief overview. I'm just going to talk briefly about salmonellosis and our data pertaining to that, the method principles and data sources that we use, the method itself, and then future directions that we'd like to see this approach take.

So first, salmonellosis. You're probably, in this room, familiar with salmonellosis and its role in foodborne illness. But Salmonella bacteria encompass a large group of bacteria of thousands of subtypes that are infectious to both animals and humans. Human infection with Salmonella is associated with anywhere from four days to a week of diarrhea, fever, and abdominal cramps. In most cases, it resolves without treatment. But of course, for those who are susceptible or acutely susceptible, it may lead to more severe complications. Salmonella is the most common bacterial cause of foodborne disease in the U.S. We saw earlier data from the FoodNet studies indicating 16.2 infections per 100,000 persons in our FoodNet surveillance catchment area. And if you look at the 2006 outbreak report, 52 percent of the foodborne disease outbreaks caused by bacteria were associated with Salmonella.

There is a large portion of Salmonella subtypes, and we know they are infectious to both animals and humans, and it's zoonotic. If you look at the food vehicles commonly associated with outbreaks of Salmonella, animal-origin food products are the most common cause of foodborne Salmonella infections.

So with that, we'll go into the method principle and data sources.

So our method principle was to adapt a model that was initially introduced by Tine Hald, and we call it the Danish model -- Dr. Tauxe mentioned it earlier. We wanted to take the model that estimated the expected number of human salmonellosis cases attributable to specific food commodities using the Salmonella subtype data; microbiologic data, collected at the point of processing of animal-origin food products; human illness data, and food consumption data.

So our first data source was the animal-product microbiologic data. And for this part of the model, we used USDA Food Safety and Inspection Service tests -- samples of raw beef, poultry and pork from meat and poultry processing plants as part of their verification sampling program. So this data includes the number of samples collected from the food commodity or the processing plant and then, of those samples collected, the proportion that are positive for the various Salmonella subtypes. So we did have some Salmonella sera grouping information.

However, the FSIS sampling program doesn't include eggs, and, as we know, one of our common subtypes, Salmonella Enteritidis, is commonly associated with eggs. So we felt it important to at least try to get the egg commodity also into this model. So for this, we used the Salmonella Enteritidis pilot project, which was developed initially and conducted in Pennsylvania to monitor Salmonella Enteritidis in laying flocks and eggs in Pennsylvania. So we had similar Salmonella data from this study. But this study was done in the mid-1990s.

So for our human illness data, you're probably familiar already, we've spoken about several of these data sources already. But we use the National Salmonella Surveillance System for our human illness data. So this collects laboratory-confirmed data from all 50 states, and again, thinking of the pyramid that we saw earlier, it depends on a person who becomes ill with Salmonella seeking health care, having a lab test submitted, having the Salmonella confirmed in that lab test and submitted to the CDC surveillance system. And so the report itself, because it's laboratory-based and they're just submitting isolates, obviously does not actually include the vehicle or the source of exposure associated with that isolate. We didn't have any information, epidemiologic information, informing that isolate.

So for that, we used FoodNet information. Again, FoodNet does active surveillance for laboratory-confirmed Salmonella. And in this case, although not every FoodNet case or Salmonella case ascertained in the FoodNet catchment area has a complete epidemiologic database accompanying it, there is information accompanying that case about whether it was outbreak-related or travel-related. And for this study, we were really interested in estimating the sporadic portion, as we heard earlier -- so those diseases that were not associated with outbreaks and not associated with travel, to try to get at the illness associated with sporadic cases attributable to domestic consumption of animal-based products. And then our last data source – we needed food consumption data. And for this, we used the USDA Economic Research Service Food Availability and Consumption Data. And this database collects data on food availability of the different commodities and subtypes and then estimates the losses associated with that and is a good proxy for consumption of the various food commodities included in our model.

So those are our data sources. So what did we do with them, and what was our method? So I have a very colorful mathematical equation here,and I've highlighted each variable in its own color to help us track it here as we go through the model. So the lambda portion, or the far right part of the equation, represents the number of observed Salmonella illnesses. And so the model assumes that this is a multiplicative function of the amount of the animal commodity or the animal product consumed. So that would be the M term here, is the amount of the commodity actually consumed, times P, or the prevalence of the particular Salmonella serotype in that commodity, based upon the USDA FSIS data.

And then we have these multiplicative factors, A and Q. And these represent relatively to each other. The A is the probability or the suitability, if you will, of a particular food commodity being a transmission vehicle for Salmonella and causing human illnesses. So this is a multiplicative kind of weighting factor that compares the various food commodities to each other on how likely they are to be vehicles of salmonellosis for foodborne illness.

The Q factor is Salmonella serotype-specific. So for each Salmonella type -- so Salmonella nteritidis would have a different Q than the Salmonella Typhimurium, for example. They're two different Salmonella subtypes. For each Salmonella subtype, we have this factor Q, which is a relative indicator of how likely or how probable that Salmonella subtype is to cause human illness. So it's sort of a proxy for virulence, or proxy for infectiousness of that particular Salmonella subtype compared to the other Salmonella subtypes in our model.

So then recall our human illness data sources. We had the one human illness data source, which is our laboratory-confirmed illnesses with no epidemiologic data. And then we have the FoodNet data, which gives us a subset of that sample, but where they have determined outbreak versus travel-related. And so this table shows on the FoodNet side the matrix of possibilities a particular Salmonella isolate could be assigned to. So within a FoodNet site, a Salmonella isolate could be in the box saying, yes, it was international travel-related, and yes, it happened to be part of an outbreak. It could be, no, it wasn't international travel-related, and yes, it was part of an outbreak. Or the information could be unknown as to whether it was international travel-related or not.

So you can see, going through the table, all the matrix of possibilities. And what we were interested in for this was the sporadic box, so those - - that proportion of Salmonella illnesses that were not related to international travel and not related to an outbreak. But of course, we have these cases that fall into one of the unknowns boxes, and so for that we use some mathematical modeling to try to estimate, based upon what we knew, the known portion of the box, we estimated the number of cases that would have fallen into each of those unknown portions so that we could add a portion of the unknown that would likely fit our definition of sporadic Salmonella cases.

I'm getting ahead of myself. There we go, back to our method. So back to our little formula. So as you can see, we used a variety of mathematical techniques. We used Bayesian analysis, a Markov Chain Monte Carlo simulation to estimate our model outputs. And this is a very sophisticated modeling processes, because it's a very complicated process. We were initially serotyping or initially had hundreds of Salmonella serotypes. Each one has its own Q value that we're trying to calculate with this model. And even though I make it sound quite simple here in my bullet points, in that our A and Q multipliers were estimated using our known values for the other three, again, each one -- trying to calculate a Q value for each one -- was very complicated. And so this was, this -- I'm talking today about an iterative process where we had to go back and adjust the model and adjust the model iteratively to try to see how many subtypes we can include in a model and get some reasonable estimates.

And this was done -- the model kind of runs simultaneously. So we're simultaneously estimating all of these Q values and A values and the proportion of illnesses attributable to the different food commodities simultaneously. So obviously it's a fairly statistically robust process that we're trying to use.

So our model output, then, that we're looking for, then, of course, is the number of cases attributable to each food product and for each year that we used in the study. And our study was encompassing 1998 to 2003.

So what our goal is for this model, again, what I'm presenting today, represents our process, really, of adapting what was basically -- what seemed to be a straightforward model developed in Denmark for their data sources to our data sources in the U.S. Again, their data sources were different. They have different surveillance systems in their food commodities and that sort of thing. So we had to do this processing, in partnership with FSIS, of identifying the data sources and adapting their model and their process to our data sources. So what we -- the nice thing about the model and the outputs that we get from it is that we get an estimate of the Salmonella -- attribution of salmonellosis across our animal-product food groups, based upon food consumption patterns, our ERS data, and the frequency of the various Salmonella serotypes as measured in the actual food commodities.

We also get, through our A values, a relative ranking of the food commodities and their likelihood to be a vehicle of transmission, and, likewise, we get a relative ranking of Q values across our Salmonella subtypes of how likely they are to cause human illness. So we're incorporating a factor for both the food vehicle and that transmission pathway and also the Salmonella serotype and its infectiousness.

So, as I said a couple of times, this was an iterative process. And so we learned a lot, and we have some future goals for how we would like to use this model in the future. For one, we'd like to try to adapt it to new consumption data and see if we can get more specific consumption data included in this model and refine our estimates. We'd also, ideally, like to include additional bacterial data, so if we could get some molecular data looking at drilling down to finer detail, we would feel more confident in associating our Salmonella data that we find in the animal product with our human illnesses. This adapted model ideally can link our pathogen data that we're getting in our food commodities to estimates of human illness, and then contribute ultimately what we're all here talking about, the strategies for food safety improvement.

So this was -- I'd like to recognize some of our contributors and advisors. This was a collaborative project of CDC and the FSIS, and represented a broad range of inputs, expertise, and including the original developers and those who continue to work on the Danish model in Denmark. And hopefully we still have some time for questions. Thank you. Applause)

DR. BRADEN: Thank you, Dr. Cole. Well, now you are all experts in the CDC-specific sources and efforts in providing some metrics for food safety. We do have some time for some questions, and I see that there one or two microphones are being placed in the center aisle. So if you do have questions, I do request that you come to the microphone and identify yourself, and, if possible, who among our panelists would be best to answer.

DR. MACZKA: I'd like to ask a question of Patricia. Carol Maczka, FSIS. Thank you very much; your talk was very good. You mentioned that you used the Mead 1999 paper and that I guess you're waiting for the new burden estimates in the paper that's undergoing revision right now. But why didn't you just use the FoodNet data and get the case rates from that to get the burden estimates? Why can't we just do that?

DR. GRIFFIN: I didn't say that I used the Mead estimate for the new attribution estimates that we're working on. I just gave that as a hypothetical example to illustrate the methods. And what we plan to do is use the new burden estimates.

DR. MACZKA: But why not just go to FoodNet and get the case rate from that? Or do you have to wait for that?

DR. GRIFFIN: Right. Because in outbreaks, we have many more pathogens than FoodNet tracks. For example, many outbreaks are due to Clostridium perfrigens, and FoodNet doesn't collect data on Clostridium perfrigens. Similarly, for a number of other pathogens. So FoodNet tracks a number of pathogens, but if you want to look at the whole range of pathogens that we see in outbreaks, then you need other data sources.

DR. MACZKA: So you just couldn't take the FoodNet case rate data, times it by the U.S. population, and then adjust it by the adjustment factor for sporadic illnesses to get the total illness number?

DR. GRIFFIN: We're working on new burden of illness estimates, and you heard how we're doing them. And we use the FoodNet data for the pathogens for which FoodNet does surveillance, and it's a very -- as you heard, a very complicated formula. We delete the ones that we think are due to foreign travel, so that we're only looking at domestically acquired. We apply a series of multipliers based on whether people seek care, and on severity of the illnesses. So it's actually a very, very labor-intensive process. It's not something that we can just sort of say, oh, let's say, here's the number from FoodNet. Let's do a couple of multipliers and then we'll just use that.

Dr. Scallan presents things so they sound pretty simple. What she doesn't tell you is that the amount of time that it took her, working on this full-time, was about a year and a half, you know, if she worked 40 hours a week every week and didn't take a vacation. And she was doing about half the work. Mike Hoekstra, who is also here, spent a similar amount of time, and a number of the rest of us also spent a lot of time. So it's not something where you can just get a number and just do some multipliers. It's very, very labor intensive.

MS. SMITH DEWAAL: Thanks. Caroline Smith DeWaal, CSPI. Really excellent presentations, and I think this work is really exciting.

I wanted to ask a question of Dr. Scallan regarding the norovirus figure. I think you said, and if I didn't quite catch it, I apologize. But about 11 percent of the unknown cases are estimated to be norovirus. And you're basing that on a European estimate?

DR. SCALLAN: So, that's not exactly what I said, but I'll explain, kind of, the norovirus figure and where it comes from. Certainly I think in the lists of things to do and the data gaps and the things that we're going to focus on for next time, norovirus is definitely out there. We really don't have any routine surveillance data for norovirus. Thankfully, eFORS, which is the data that I've included in our study, which is the Electronic Foodborne Outbreak Reporting System, has now transferred to the next generation, which is NORS, the National Outbreak Reporting System, and that is capturing norovirus outbreaks. So at the very least in our next iteration, we will have some outbreak data on norovirus. But the data that I have to deal with and for the time that I was looking at, there was no surveillance or outbreak data for norovirus. But we want to include norovirus. We think it's an important contributor to foodborne illness.

MS. DEWAAL: Yes. DR. SCALLAN: So the only way that we could do this was to use information that was available from other countries. And that's where this 11 percent comes from, is studies conducted in the U.K. and in the Netherlands and Australia, where they estimated what proportion of acute gastroenteritis is due to norovirus.

So obviously, huge limitations with what. You know, we're assuming that studies done in other countries have some bearing on what we're doing here, and we're applying this 11 percent to our estimate of acute gastroenteritis, which is this kind of starting with the bottom and working up. Whereas what I think we would prefer to do is start like we have with most of the pathogens, start at the top and work down.

MS. DEWAAL: Yeah. The reason I'm asking is because we're starting to break out, based on your new datasets that are available -- thank you very much -- we're starting to break it out by States. And where norovirus is being monitored in the States, I mean, I'm seeing some variation, but you know, it could be 25 percent of outbreak reports where they are fully investigated or fully attributed, which is the dataset that Dr. Griffin is also working with. So I think that 11 percent may be an underestimate, and I understand you're at the end of this process, but it may be unfortunate because I think that number will be coming up when you start looking at State data.

DR. SCALLAN: Yeah, norovirus is definitely a challenging pathogen to deal with. Because, like we showed for our pyramid, to be identified as norovirus, people have to seek care and submit a stool sample, and the lab has to test. And because there is so much norovirus, there's not that much testing that goes on. So the picture is often kind of clouded by what cases are followed up on and what laboratory testing is done. So it's very much on our agenda for the next iteration.

MR. BLACKWELL: Steve Blackwell for EHA Consulting. I just was curious about how the FoodNet sites were selected initially, and is there any chance or any possibility of them increasing the number of sites to get better data?

DR. GRIFFIN: So, many of the FoodNet sites were part of a system that had been started by another group at CDC before CDC began the emerging infections program. And those sites were sort of grandfathered in, and after that, there was a process for several years to admit new sites. And although people wanted to look at representativeness by the population and many of those other things, the major criterion was really, was this site capable of doing it? Did they have, you know, the people who we thought could – and it wasn't just us, there are other groups at CDC who are involved in these sites -- could they get good data that would be trustworthy? Because it's sort of one bad apple ruins the bunch. If you're pooling data, you want to have the data from all sites to be very reliable.

And so eventually, 10 sites were selected. And in fact, when they got to 10, it sort of broke the bank, and 10 felt like more than people could handle. And it's been hard on the sites, because the funds have been similar for many years, and so they're trying to do the same thing without increased funding, so that there had been talk for the past several years of actually decreasing the number of sites, although it would be nice to increase them.

MR. BLACKWELL: Yeah, it just seems to me that -- a quick comment -- that at every level of these modeling, a lot of assumptions are made. And when you build assumptions on assumptions on assumptions, the final number really is very difficult to pin down, so. DR. GRIFFIN: Yeah.

MS. HOFFMANN: Sandra Hoffmann from Resources for the Future, and it's really a follow-up to the prior question, in a sense. I've got a question that I know none of us, as researchers, really like to get this kind of question. But we're thinking today about performance measures. So what I'm hearing is that we would like, as a performance measure, something that we can view as changing over time so that we can track performance. In terms of hard epidemiological data, it seems to me what we really have that changes over time is FoodNet data, and that's for a limited number of pathogens. We could take that and we could apply food attribution numbers to that to see trends in food over time for those pathogens, possibly. I'd like your thoughts on that. But I think even more broadly, because of -- we know that we'll perform to what we look for, and we're looking for a limited number of pathogens -- I'd just like your thinking on, okay, what do we need to do, what we can do, what's feasible, for trying to get performance measures that are actually going to help us improve overall health from foodborne illness?

As I say, not a question that we like to get, as researchers, but you all know more about this than probably any of us here, so ....

DR. GRIFFIN: I think several of us are going to make several comments. One thing, the direction I thought you were going, which may not be what you want, is that FoodNet tracks a certain number of pathogens. And there are others that FoodNet doesn't track, because clinical laboratories just don't test people for those agents, so we don't have a way of measuring sporadic illness due to those agents.

But we can measure through outbreaks. And we know, when you look at that map that Dr. Mahon showed, and those dark green states that investigate and report a lot of outbreaks compared with the white and the light green states that don't investigate or report many, we know that there's a lot of outbreaks out there that aren't getting detected, or maybe they're detected and the states don't have the resources to investigate them. So there are a lot of people getting sick, and their illness is being wasted, because there's information that we all could get to figure out what's going wrong.

And so if we can get more of those outbreaks investigated and reported, then we would learn a lot more about what agents and what foods are causing illness, and then we would apply that in these models. But I know Dr. Tauxe has something to say, as well.

DR. TAUXE: It's a great question, and one that we think about a lot. I think one is, we're quite cognizant that between 1999 and the last burden estimate and hopefully later this year, there's quite a stretch of time, and enough time for many different variants to occur, so that that does not let us easily compare.

But going forward, I think that having estimates that come out at more frequent intervals, more regularly, and combining that with attribution measures that can come out at some more frequent intervals than whatever we would be able to achieve right now would help.

That's all dependent on the activities of our clinical system, our healthcare system, and our public health system. And one thing that's been explored that we find really intriguing in several European countries now is actually independent of that -- independent of those systems. And that is beginning to look at serological -- that is, looking at antibodies in people's blood for a sample of the population, to what they've been infected with. And as those assays are worked out and applied to national collections of serum, it is now becoming possible in, say, Denmark or Holland, to say what proportion of the population has had Campylobacter, what proportion of the population has had Salmonella within some period of time. And those are very interesting kinds of measures, and if that could be developed for other pathogens as well, that becomes a way of looking at that.

That is the way we track Toxoplasma in this country. And there is actually progress being made with Toxoplasma, for which we've got really otherwise no surveillance at all.

So my answer is, we could do better in coming out with estimates more frequently, and there are laboratory approaches that might allow us to sort of come at this from a different angle, as well. DR. BRADEN: We'll take one more question -- DR. GRIFFIN: Chris, I think some of us had more comments on that.

DR. BRADEN: You want to -- okay. I was trying to get as many questions in, but we're all -- so ...

DR. MAHON: So just quickly, then. We've talked about a lot of different modeling approaches today, but one that we didn't mention is actually some work to try to blend together the attribution information that we learn from outbreaks. So we get really, I think, a very good and very timely sense of what's going on from outbreak reports. And to the extent that those reports are complete, they offer wonderful information on what's happening.

And the modeling -- the additional modeling effort is to blend that with the information that we can get from things like case/control studies and FoodNet to try to get an overall sense of what's going on that can be tracked over time.

DR. GRIFFIN: So you asked about ideas of measures of progress. And we now, in FoodNet, have measures of progress. We track the incidence and we make goals.

We don't have that sort of thing for commodities. And once we get our attribution model worked out and figure it out, that's something that I think everyone will want to look at: can we make goals for decreasing the number of illnesses due to particular commodities? And we have to think hard about that, for several reasons. For example, we may want to decrease illnesses due to commodities that were contaminated at the source. And we have to think about the methods for doing it, as well. You saw in my slides that you quickly lose a lot of outbreak data, and even though we get 1,200 outbreaks reported a year, to really have robust data, we need a lot of outbreaks to analyze.

But I think that's one thing that everyone would love to see, is to do a trend analysis on outbreaks due to certain commodities. But we have a lot of methods to build to be able to do that.

DR. BRADEN: Okay. I'm going to have to draw the session to a close, because we're eating into our lunch hour. Please be back to resume at 2:00, and if you have any other questions, remember that we will have some time later today to address them. Thank you very much. Applause) (Lunch break)

MR. GUZEWICH: Please take your seats. We want to stay on schedule so we have ample time for the question and answer period at the end of this session and the one that follows. I realize that a lot of people probably are still coming back from lunch, but we're going to get started anyway.

My name is Jack Guzewich. I work for the Food and Drug Administration's Center for Food Safety and Applied Nutrition in College Park, Maryland. I'm the moderator for this session.

The sessions you heard this morning, the CDC session in particular, was talking about human health-related indicators of food safety progress. This session is going to have some different perspectives on food safety progress as seen from different kinds of regulatory agencies -- the Food Safety Inspection Service at USDA and the Food and Drug Administration, on the retail program, and then a New York State speaker talking about how they link outcomes with their regulatory program. Then the Grocery Manufacturers'Association, an industry perspective on food safety metrics, and then a consumer representative on consumer perspective on food safety metrics.

Our first presenters this afternoon, and we're going to try to stay on 15 minutes per session here so we have time for Q&A at the end - our first presentation is going to be on measuring progress through foodborne illness attribution and pathogen verification testing. Our two speakers in this time slot are Erin Dreyling, who is the Director of the Data Analysis and Integration Group in the Office of Data Integration and Food Protection at the U.S. Food Safety and Inspection Service of the Department of Agriculture. Dr. Dreyling is the project lead for the agency's development of data-driven improvements for processing and slaughter inspection. She has led the development of FSIS predictive analytics component of its new information infrastructure, which is intended to improve the agency's ability to more effectively use data to inform policy development and predict and prevent public health risks. Previously, Dr. Dreyling served as the Deputy Director and Analyst at the DAIG, and her co-speaker is Christopher Alvares, who is the Deputy in that same unit. And his Data Analysis and Integration Group provides data analysis and reporting support to FSIS and is involved in the development of the agency's new public health information system. He is also Acting Branch Chief of the Field Operations Analysis Branch in the Data Analysis and Integration Group. The Field Operations Branch provides data analysis and reporting support to the agency's 15 inspection districts. Previously, he worked in the biotech industry on generation and analysis of data.

And I believe Dr. Dreyling is speaking first, is that correct? Dr. Dreyling.

DR. DREYLING: Okay. Good afternoon. I'm Erin Dreyling, and as I was introduced, I'm the Director of the agency's Data Analysis and Integration Group. And my colleague, Chris Alvares, and I this afternoon are going to share our presentation. And we're going to talk with you about how FSIS is using attribution and our pathogen verification testing results to measure progress at the agency.

So first I'd like to talk with everyone about why FSIS needs attribution estimates. We talked a lot this morning about what foodborne illness attribution estimates are, and I'd like to talk, from a regulatory agency perspective, about why we need foodborne illness attribution.

First, we need foodborne illness attribution estimates for the products that FSIS regulates, which are meat, poultry, and processed egg products, to determine which food items are the major sources of human illness. We want to use our attribution estimates to estimate the number of illnesses that are caused from the products that we regulate, and we want to use those illness estimates to measure our progress in terms of established public health goals such as the Healthy People 2010 goals and in the future, the Healthy People 2020 goal.

I'd first like to review FSIS' proposed methodology for foodborne illness attribution estimates. And these draw off a lot of the work that CDC has presented already to us this morning. We are working very closely with CDC and FDA through the Food Safety Working Group to develop collaborative methods for attribution, and we've applied their methods to measure performance for FSIS. And here's what we have done so far.

First, we are using CDC outbreak data, as we discussed this morning. We are looking at seven years' worth of data from 2001 through 2007. We first separate the outbreaks into complex food outbreaks and also single food outbreaks, and we then estimate attribution for our simple food products. We then go back, and using the Painter methodology that was discussed this morning, we estimate foodborne illness attribution for complex food products, and we then derive a total foodborne illness attribution estimate for our products.

We then use our attribution estimates to determine the number of foodborne illnesses that would be attributed to FSIS-regulated products. And what I've shown you here is an example. So you can see, on the far right, we have the estimated total illnesses that we would derive from CDC's FoodNet data, the 2008 data that is the most recently available estimate from FoodNet. We would derive a case rate from the 2008 FoodNet data, which in this case we have shown Salmonella. That was 16.2 cases per 100,000. We then multiply that by the population for the United States, which is 303.8 million, and we apply a scaling factor for unreported cases, which is derived from the Mead 1999 paper, which is a scaling factor of 38.

That would give us a total illness for Salmonella of 1.87 million from all food products. We then would apply FSIS' attribution estimate for the products it regulates -- meat, poultry, and processed egg products -- and we would then have an illness estimate for only FSIS-regulated products.

So what we have shown on the next slide is a more detailed description. Again, our methodology, we take the FoodNet 2008 case rate. We look at a case rate specific to a pathogen of interest. So we could look at Salmonella, E. coli , or Listeria . We then follow the methodology of multiplying by the population, and a scaling factor for unreported cases, and we get an illness estimate for each pathogen. We then apply our attribution estimate for FSIS-regulated products that we derived from the outbreak data, and we can break that down into five major food categories of interest to FSIS, based upon the outbreak data.

And we have shown you here an example for E.coli. You can see that we have attribution estimates shown for beef products, poultry products, pork products, and ready-to-eat products, and a total for all FSIS products for attribution. So we take our FoodNet-derived illnesses. We multiply that by the attribution specific to FSIS products. And again, you see, on the far right, we have an example of the estimated number of illnesses specific here for E. coli from FSIS products.

What we've shown on our next slide is the illnesses for FSIS products that we have derived for three pathogens of interest to the agency. And these are specific to Salmonella, E. coli, and Listeria. What I've shown is the 2008 total illness estimates for these pathogens, the FSIS attribution estimates that we have derived specific to our products, and then the total number of illnesses that we would attribute to our products.

So I said that we needed attribution estimates and illness estimates to measure our performance, and I'd like to talk briefly with you this afternoon about how FSIS is using its illness estimates to measure its performance. We want to measure our performance in terms of established public health goals, such as Healthy People 2010. And so what we have done is to take the Healthy People 2010 goals that were established for Salmonella illnesses, E. coli illnesses, and Listeria illness. And these were just general to Salmonella, E. coli, and Listeria. They were not product-specific. We have taken our attribution estimates, and we have derived the portion of the Healthy People 2010 goal that we feel our products are responsible for. And we can use this to then measure our performance in terms of these goals to see how our products are measuring up.

In order to measure ourselves in terms of these FSIS product-specific goals, we use our laboratory testing verification data. And Chris is going to talk in just a minute about how we are working to improve those data in order to use them to estimate exposure. But here is the method that we propose to use.

What we would do is, on a quarterly basis, in order to measure our progress in terms of goals, such as the Healthy People 2010 goals, we would take our laboratory volume-adjusted percent-positive rate, and we would divide that rate by the volume-adjusted percent-positive rate for the prior year. And we would multiply that by the illnesses that we would estimate for that prior year. And what we would get by using that ratio and that illness number is the number of illnesses we would expect in the present year, or the present quarter. And that gives us a way to measure our progress in terms of public health goals that are expressed as illnesses, such as Healthy People 2010.

I do want to let everyone know that we are actively working through the Food Safety Working Group on our attribution approaches with FDA and CDC. We are coordinating on first focusing on developing attribution methods for Salmonella Enteriditis, and that plays off of the work that Carol talked about this morning with regard to having specific metrics for Salmonella Enteriditis. We are also working to derive uncertainty estimates around our attribution work, and also we are looking for methods that will allow us to include serotype information in our attribution estimates.

So what I would like to do now is let Chris Alvares come up and talk about how we are looking to improve our laboratory verification testing for the purposes of estimating public health outcomes. Thanks. MR. ALVARES: Thanks, Erin. So as Erin mentioned, and really several before me, we -- FSIS is very interested in and focused on reducing illnesses due to FSIS regulated products. And in order to measure our progress in that area, we have been utilizing our verification testing data to estimate the levels of contaminated product that really is in the food supply. But we recognize and acknowledge that there are some problems with the design of those sampling programs and their application in terms of estimating prevalence, and I'm going to talk a little bit today about some of the ideas we have and the work that we're doing to really improve those programs or if necessary, initiate some new programs that would really be better suited towards this goal.

The things I'm going to talk about today are really proposals at this point. These aren't things that have been decided upon or are in place at the moment, but it does reflect our current thinking and our plans for future directions for these programs.

So everyone's, I think, talked a little bit about the importance of metrics and prevalence in particular. It's necessary for quantitating risk and understanding how much of the food supply might be contaminated, measuring trends, as well. We've talked about the importance of getting timely estimates. FSIS is trying to develop ways of reporting those trends on a quarterly basis, and along those lines, as well, measuring performance as an agency, not just in terms of sampling, but really, how are the impacts of all our policies and our inspection procedures impacting the safety of the food and ultimately, really, to drive reduction in the contamination rates and to inform policy as well as to potential courses of action that they could take.

So again, we are linking our performance to the Healthy People 2010 and the 2020 goals. We take the human case rates that are set by that, and a proportion of that is attributed to FSIS products. Those products, or that proportion is then translated into volume-weighted percent positive rates. I have a description here for E. coli in particular, but we do this separately for each of our three major pathogens, Salmonella and Listeria included. And so we have set annual and quarterly goals for meeting these volume-weighted percent-positive rates, and that's some of the metrics that we're using to measure our performance.

As I've said, we recognize that there are some limitations in the design of these programs. Our major sampling programs were really designed originally to be verification sampling programs directed at the establishment to verify their HACCP plans. But we do, and we have been using them to get our perspective on the industry, as an industry, –how the product's doing as far as contamination rates. For E. coli, we're talking about raw ground beef. For Salmonella, broilers, and we're expanding that to all raw products. And Listeria, in particular for ready-to-eat products, as well.

We do recognize that there -- we would like to get more samples, increase the precision of our estimates. This is important for making data-driven decisions, having reasonable levels of confidence and certainty in our measures. There was some discussion earlier about uncertainty and confidence intervals, and I think that's important in the measurements and the reporting that we do, as well.

And we also recognize that there's a need to identify and address some sources of bias in our programs. Particularly, there are some exclusion criteria, some performance-based sampling, that may be biasing some of our results. And we're aware that there's some concern about the announcement issue, which briefly is simply that the plants, the establishments being sampled, maybe be forewarned that they're getting a sample, and there's a potential there to take steps to ensure that they pass that test or that set of tests. Or on the converse, that they may be going in a period of time where they know that they won't be sampled, and that may be -- that may lead to temptation to relax standards or take other steps.

So we want to be able to address these. Really, when the agency talks about its measures of prevalence or estimated prevalence, historically that's been based on our baseline studies that we conduct. And those have really been recognized as our measurements for prevalence estimates. We use those for setting performance guidance criteria for industry. So we really took a look at these baseline studies and said, how do these compare to our prevalence programs? What can we do to better design those programs to be more baseline-like in nature?

So I'll move quickly through some of this. But we are working toward this concept of ongoing baselines, which is basically very similar in design to the traditional baselines that we do. They are statistically based; they are inclusive of all establishments. They have randomization included in the selection process. They will be done continuously, as opposed to the traditional baselines, which are really done for a period of time, and then there's an extended period where they may not be performed.

So I have just a brief comparison here. We're not talking about ending traditional baselines. They still have a major role to play in the agency. But there are some differences in how they'll be applied, and it really has to do with measuring existing commodities versus new commodities, measuring existing pathogens versus new pathogens that we have.

So I've been taking a look at our sampling programs for E. coli, Salmonella, and Listeria. These are the three major areas that we measure contamination rates in. And I'll talk just briefly about each program.

We recognize that there are issues with all of them, and they all are designed a little bit differently. For the raw ground beef program, there are some issues around notification. But really we feel that that's fairly similarly designed to our baseline concept. But there are sample size issues that we want to address, and so we're looking to maybe increase the number of samples collected to -- by about maybe 50 percent, given that we can get the available resources and work out details there.

In terms of Listeria, it's a little bit more complex. There are more food products that are sampled. We actually have that program split into two. We're looking at trying to unify these two programs to really pool the amount of data and get the most value out of those data. So again there, we're looking at making some changes to those programs to make them more baseline-like in nature.

We would still address the fact that some of the products, we believe, are higher risk than others, and we would sample that in an appropriate way to focus more of our sampling on those higher-risk products, but in this stratified or random probabilistic way.

Finally, with Salmonella, there are several sources of bias that we've identified in the design of this program. Really, we're focused on the exclusion periods that exist or criteria that exist for certain establishments. Some of this is performance-based; some of it is volume-based. There's also sort of a non-random nature to the sampling. Our Salmonella sampling is done in sets. These are really consecutive operational days, and then there's an extended period where if the establishment reaches a certain level of performance, they're not sampled. And so we want to address that. There are some issues with making changes to the current program, and so we're looking to initiate a new program that would be more baseline-like in nature, sample the raw product commodities for Salmonella, and give us potentially some better estimates of product prevalence.

So in terms of next steps, as I said, these are really proposals that the agency is considering and future directions that we're moving towards. We need to work out specific proposals. There are a lot of details that need to be addressed. There are policy impacts. There's resource balancing that needs to be done. We are working towards having some concrete actions taken to implement in FY2011. We really believe this will be an improvement on the agency's measurements, on the agency's data, and our ability to measure prevalence, and make decisions based on that. Okay. Thank you. (Applause)

MR. GUZEWICH: Thank you, Chris and Erin. Our next speaker is Kevin Smith. He will be speaking about the FDA data on a retail food survey. Kevin is a Director of the Retail Food and Cooperative Programs Coordination Staff in FDA's Center for Food Safety and Applied Nutrition. He directs FDA's Retail Food Protection Program, which operates as a cooperative program with state, local, and tribal food protection programs. His staff is responsible for development of the FDA food code and fostering cooperation between FDA and its public health and industry partners to improve food safety in the retail environment and in FDA's Grade A milk and mollusk and shellfish programs.

Previously, he was a Project Manager for Standards Development and Environmental Technology Verification at NSF International and began his career as a sanitarian in New York State. Kevin Smith.

MR. SMITH: Good afternoon, everyone. I hope everyone's feeling good after lunch, and I'll try to keep you alert and awake.

As Jack said, I'm going to tell you a little bit about what we refer to generally as FDA's Retail Risk Factor Study. It is essentially an effort by the agency to measure the performance or evaluate the performance of the overall retail food safety program in this country. I will try to quickly give you a little flavor for what the study was all about, what the objectives were, a little taste of some results, though I'm not really here to tell you about the findings of the study, but just to give you an overview of what it was all about. And some of the challenges and limitations that we faced, and perhaps some ideas that we're looking to in the future.

This effort was initiated back in the mid-'90s, '96, '97. And, if we were doing it today, and with the benefit of all the discussion that's already happened here today and the recent work among all our partners, who knows, it might be a different design altogether. But I think it was pretty groundbreaking for the Agency at the time, and the fact that it started back in -- the first data collection was in '98 and is still viable today, and we're awaiting publishing the results shortly on the most recent data collection -- tells us that there is value to what we are learning from this study.

Briefly, there are a number of objectives we're trying to accomplish with this study. And that made it a large and somewhat complicated study. Perhaps if it was a little more focused, maybe we would be able to do more data collection. But what we attempted to do was assess the compliance across the nation with controlled measures that are in place to control risk factors in retail and food service facilities of all types. The purpose was essentially to use that compliance information as a performance metric for the National Retail Food Protection System, and that system, probably everyone in this room knows, perhaps, that it was very large industry, massive industry, a million and a half establishments, with the regulatory authority lying in a partnership between federal, state, local, tribal agencies, with the direct regulatory authority lying largely with the state and local. So FDA has a very supportive role, but we are not the direct regulatory authority.

What we hope to do is establish a national baseline. An initial data collection, establish a baseline, and then periodically assess whether practices in retail establishments of all types have been improving or regressing. So that's what we set out to do. We also, with that established, have a strategic plan for our national retail team; that there were goals set, like a 25 percent reduction in the occurrence of out-of-control risk factors. That was a target that we set for ourselves, and this study is helping to see how we're progressing on that target. That target was set for 2010, and that was built into the Healthy People 2010 initiative, which some folks have mentioned before.

Other objectives – there are a lot of them, but we also wanted to establish a model for how our state and local partners could assess their own performance in their own jurisdictions. Are establishments that they regulate, how are they doing in terms of controlling the key risk factors that result the most often in foodborne illness?

We also had a programmatic purpose, as well. We knew that these -- when we published these data, it wasn't just to get the data. It was to actually inform folks, both the regulatory and the industry, where they should be paying priority attention. Which risk factors, and I'll get into what those are, which ones need attention? Which ones do we need to focus our efforts on to control?

There was not an attempt to correlate the findings of this study with the actual incidence of foodborne illness. Although we played with the idea, and it would have been nice, and maybe in the future we'll consider that. But we did not attempt to make a connection between what we were seeing as far as illness or even - - we didn't do any sampling of foods to see about whether it was actually creating a hazard in the food itself, either. We're just looking at the practices themselves.

It involved a large data collection effort, and I really probably should have said this ahead of time. There are three data collection periods, 1998, 2003, and 2008. Each time, roughly 900 establishments visited, and an observational study conducted where you observe the facilities for compliance with key provisions of the FDA Food Code. We use the 1997 Food Code, even though it's been updated many time since. We used the '97 Code as the standard of measurement, not the individual prevailing public health codes that are in place at the state and local jurisdictions. Instead, we want to have a single standard of measure. That was the '97 Code. For those who aren't really familiar with the Food Code, it contains a lot of very detailed control measures that retail food establishments, food service establishments, are required to have in place. The states and locals adopt the Code. The Code's a model for their regulations.

So the expectations of what the industry, what the retail and food service industry, is supposed to do are pretty clear. We just want to assess how are they doing in achieving those -- in conforming to those requirements.

This is actually nine different studies, even though it has -- the reports are issued under a single cover, but it's really nine separate studies, because we break it down by these facility types you see here. Three institutional food service types, two restaurant types, and four retail food store types. And the data are all reported separately, because we wanted to call attention to the risk factors that those individual facility types should be focused on. It might not do the operator of a nursing home much good to know how well fast-food restaurants have proper cooking temperatures under control, so we wanted to present the data individually for those facility types. It makes for a large study.

We focused the study on data collection around five key operational risk factors, the term we use.

They're derived from the contributing factors that CDC routinely reports on. There's a pretty good understanding of when outbreaks occur as a result of practices that are taking place in a retail or a food service establishment, what typically those contributing factors are. We use those contributing factors to identify the risk -- what we are calling risk factors, because it's before the outbreak happens, it's not after. We have those risk factors, and under each of those risk factors, there are anywhere between five and a dozen individual data items that are specific to Food Code provisions that roll up into those risk factors. You can see those: Food from unsafe sources, inadequate cooking, improper cold holding and hot holding temperatures, contaminated equipment or causing contamination, and poor personal hygiene. So all of those things, if every establishment did everything right in those areas, we would eliminate most of the contamination, and proliferation hazards that are out there.

We also did a little bit of data collection on some chemical exposures, proper use of chemicals and storage of chemicals, in the facility as well. That's kind of in the "other" category.

So the findings are reported in a -- the '98 data were reported in a 2000 report. It's available on our website. The 2003 data are reported in a 2004 report that's available on FDA's website. And now we have -- we collected the data in 2008. We have a report on that, and that has not issued yet, because we're holding onto that to be issued concurrently with a trend analysis report that will be coming out shortly that will attempt to see, did we see a trend across those three data collection periods.

So on the stand-alone reports, there are a lot of data in there, But kind of the highlights are that we report out-of-compliance percentage for each risk factor, and then we also report the out-of-compliance percentages for each individual item that rolled up into those risk factors. And then we present the in-compliance rates for all of the 42 data items that make up the collection into a single number for each facility type, a single in-compliance figure. That relates to our performance goals.

All right, so as an example of the first bullet, the percentages for the risk factors, this is from the 2009 -- 2008 data collection, the report that will be issued shortly. I just picked one randomly. This happens to be the deli facility type. And there are data like this for all the different facility types. It's a little hard to see, probably, but across the bottom you'll see each of the individual risk-factors categories, plus the Other/Chemical category. And you'll see Improper Holding Time and Temperature. A lot of these numbers are important to look at in relation to one another, because the individual number isn't as important as the comparison to other risk factors. But Improper Holding Time and Temperature, it showed delis 15 percent of the time, that you were -- the five data items that roll up into that risk factor, 15 percent of the time there was at least one observation in that facility that could -- was out of compliance with the Food Code. It doesn't mean that all the temperatures that you measured, 15 percent of them -- of the food that you measured was out of temperature. It was that in a given facility, if there was one violation of a cold holding temperature, then that facility was deemed out of compliance, and that's where those numbers figure in. Five minutes? Okay, we're doing all right.

Let me go back really quickly. You can see, Poor Personal Hygiene, 20 percent. So out of the – I think there are -- well, Personal Hygiene has five data items that roll up into it. So 20 percent of the time, you would find one of -- of all those data items, 20 percent of the time there was an out-of-compliance observed. I'll expand a little bit on what poor personal hygiene includes. That has proper adequate hand washing, that you have the proper facilities, that you have good hygienic practice, that you have the hand washing – there are a couple different about the hand washing facilities -- also preventing contamination from direct contact with hands. All of those things roll up into that one poor personal hygiene risk factor, but in this case, you can see, in 52 percent of the delis, there was an observation at least once where improper hand washing was conducted. That doesn't mean that 50 percent of the staff or 50 percent of the time that individual establishment wasn't washing their hands. It means in 50 percent of the establishments, you're able to see at least once incidence where someone didn't wash their hands when they should have, or didn't wash their hands properly.

So those are the kinds of data that are in the report. The trend analysis report will be of interest to folks, because it will tell us, relatively, are we seeing improvement or regression for each of those risk factors I described and each of the data items? It will break it down for each of those different facility types. You'll see data that will tell where there was a trend observed. And even though it is a big study, you had to have observations. We didn't observe the practices in every facility type. Some people aren't cooking any food for certain facility types. There aren't -- maybe weren't cold holding foods. So you couldn't get observations in every single facility you went to. That makes it tough to get enough data to have statistically valid conclusions about whether you actually see a trend over those three data points.

But to the extent we can, we have applied very robust statistical analysis to make sure that we aren't drawing conclusions that aren't there. But I think you'll -- the particular industries will be very interested in seeing the trends. I think it will inform folks at all levels of government how -- whether what we're trying to do here is making an impact or not, and where do we -- well, I can tell you that we're generally, the initial feeling is we're pretty glad that we've seen improvement in a number of areas and not regression in any significant way. So that's good. But it also will find that there's a long way to go, based on the outcomes that we observed for certain risk factors.

So just to give you a quick look at what some of the data will look like. You can't see it from here, probably, but each of the three data collection periods will be shown against one another for each of the individual data items and then rolled up in the risk factor. So that just gives you a little flavor for what it will look like. Well, this one happens to be Poor Personal Hygiene risk factor, and then the individual data items that made that up.

So, quickly, challenges -- this was a tough study to make sure that we got -- trying to represent a national system, find data that are representative nationally, is tough, because there are so many establishments and so many regulatory jurisdictions. Doing a purely random sample at the time that we got this started in the '90s really wasn't an option. It wasn't feasible from a resource standpoint, so we had to select locations where we had individuals working. So we have specialists all over the country, and they randomly selected from their communities, their locations. But there are many places around the country that – there are no data collected in Alaska, because we didn't have resources to send people to Alaska to collect the data. So that's one of the challenges we encountered.

Also, as I mentioned before, we're trying to assess trends based on just three data points. That presents challenges, but you know, I think we'll probably, if we were to transition, we might go to smaller studies more repeatedly. And certainly a single performance metric isn't going to tell you the whole story, because there are so many factors that affect retail food safety, both from the regulatory standpoint, how much oversight is there, are they doing a lot of inspections or not, do different industries have different levels of control in place? The industry is changing, there's a lot of employee turnover -- all sorts of things affect the overall performance, so a single performance metric isn't going to tell you everything, but at least it's something.

Future study areas -- I got a lot of ideas sitting here today. Listening to this morning's presenters. I'm hoping we can, in the bottom bullet, attempt to correlate some of these performance metrics, perhaps, with measures of foodborne illness metrics. That would be obviously very interesting to folks.

But we also have ideas about trying to examine the impact of individual intervention strategies. If one place has a strategy in place, how does that compare to one that doesn't? That's kind of, I think, one of the areas we'll go in. Also, perhaps, more in-depth studies of fewer practices, or perhaps fewer facility types, and do a more in-depth study of those.

So that's it in a nutshell. If you have any questions, get in touch with me. Thank you. Applause)

MR. GUZEWICH: Thank you, Kevin. We'll be taking questions at the end of this session.

The next presentation is applied food safety metrics: reducing the number of foodborne disease outbreaks by linking outbreak-contributing factors to a food safety regulatory program. It will be presented by Michael Cambridge from New York State Department of Health. Michael is a Director of the Bureau of Community Environmental Health and Food Protection for New York State. He is responsible for development and implementation of statewide policies and procedures, technical guidance, and training programs in coordination with other state and federal agencies.

Prior to the state directorship, he served at the New York State Department of Health as a District Director, Chief Sanitarian, and Health Program Administrator.

Michael Cambridge. MR. CAMBRIDGE: Okay, good afternoon. It's a pleasure to be here to speak to you on this important topic.

As Jack indicated, I'm from the New York State Department of Health. And through the State Department of Health, we work with our local health departments in investigation of foodborne illnesses. So I'll be speaking today about linking outbreak-contributing factors to a food safety regulatory program.

The chart I'm displaying now represents foodborne disease outbreaks that have been investigated and reported in New York State for the time period 1980 to 2008. A couple points I'd like to raise. As you can see with the pointer, the first couple years, '80-'81, the surveillance system was just getting underway. You know, New York State always investigated foodborne disease outbreaks, but the change was to try to understand the factors that contributed to the outbreaks to then set up an intervention strategy.

So you can also see that '82 to about '92, there are about, roughly, 138 outbreaks reported in each year. And then shortly after that, '93 to 2008, there are roughly 67 outbreaks per year, on average.

As part of the outbreak investigations, what we do is we look for, in addition to all the prevention measures that you heard, controlling and protecting the people that are affected, what we try to do is understand what exactly happened with the food item that was served. Today you heard a lot of information about doing epidemiology, linking it to a food vehicle. Now, when you find the food vehicle, what do you do? If you don't understand the food vehicle, you're not going to understand how to implement a regulatory change, or an educational change, as well.

Again, I think it was clear this morning that there are three vital players as far as governmental agencies. There's the laboratory, and yes, New York State is part of PulseNet and all the different programs you heard today, with CDC. We have mandatory reporting of foodborne pathogens and in general, if a physician suspects it's an illness that may be associated with food, it's reportable. All those cases are investigated. People are interviewed, trying to understand what took place.

But today, I'm going to focus on the environmental part. That's the part I deal with on a regular basis. We work very closely with our lab partners and our epi partners. It's just an ongoing thing.

As far as our field structure, we have about 46 local health departments, nine state district offices, four regional offices, a central office for environmental health, for epidemiology, for laboratory. And so what we need to do is to make sure everyone is working together. In addition to New York City, a large city that's part of our surveillance system, and we work very closely with them, so the information I'm going to report reflects all the hard work that's been done over the 29 years. The hard part -- well, I don't know if it's harder -- the sad part is, I started working with the Health Department in '81. So all this information, I feel like I lived it, not only had to work through it. So a lot of firsthand experience with this particular program.

I can't stress enough the teamwork, not only on the state and county levels, but also with our federal partners. Whenever we suspect we have an issue, CDC is brought in, FDA, USDA – whatever agencies would be appropriate to assist in the investigation.

So the interesting part is, right away, you know, we talked about implicating the food vehicle. So that's what we do. We go through the process as a team and we come up with a particular food item that has been implicated as the cause of the outbreak. So now, what's our next step?

You always hear about, "let's do an inspection." Well, the way we like to look at it, it's important to do a food preparation review. An inspection usually is an observation on a given date and time. You're there, what's going on in the facility. That's nice; it's important. However, your outbreak probably happened a month ago. You need to know what happened back then. So all of our training of our field staff, we focus on those kind of details. You need to understand what took place a month ago; who were the suppliers of those raw ingredients; who were the employees actively involved in the preparation of that food? Because what we're trying to do is trying to figure out the method of preparation, the significant ingredients that were part of it, any specific details that could help us identify what went wrong on that given day.

So since the early '80s, we focus on training our field staff in the HACCP principles. And so, you know, initially change is always a lot of fun. People were saying, You really can't do that, it's impossible; we have to do inspections. So there's a lot of training of the field staff, and what we like to believe, at least, is that it paid off. Because as you start looking at, following the reports that came out after the field staff understood what needed to be done, you can see the vehicles were identified on a regular basis, contributing factors people realized were an important part of the investigation.

A discussion was held today about meatloaf and those kind of foods. You know, we always look for the significant ingredient that went in there. If somebody put eggs in there, were there eggs left over? Can we test those eggs to say, you know, yes, it was the meatloaf, but it was the eggs that introduced the Salmonella.

So from the information, it's all compiled in a database. First, years ago, it obviously used to be on paper. Then we got a PC XE computer, and now, we've updated, so things are much better. But we have tracked all the information, and we're looking at the top 10 contributing factors. We follow about 17 different contributing factors, but for today's example, I just wanted to highlight the top 10.

As you can see, number one is contaminated ingredients. Number two is infected person. And number three is consumption of raw or lightly heated foods of animal origin. It's sort of interesting. That came up as a contributing factor because certain foods, people don't intend to cook. You know, they eat it raw or they just lightly cook it, like clams -- nobody -- people like them raw or they like to steam them, and so it's just lightly cooked. So we thought it was important to differentiate. So what I'd like to do is now sort of take it to the next level, all right? I selected some outbreaks that occurred during the '80s and '90s, because I wanted to show what took place then and where we stand now. So if I did something more current, then I wouldn't be able to tell you where we stand now, because it's still ongoing.

So during the '80s and early '90s, we identified a number of outbreaks that were related to ill food workers. Basically, they were identified as the cause of the outbreak. They were carrying a pathogen, introduced it to the food, and so we needed to come up with a way to identify how to prevent the next outbreak.

So we looked at the control point and said, "Well, let's just prohibit people from working when they're sick." And, well, everybody knows they're not supposed to work when they're sick, but it was happening anyway. Well, let's just wash -- "tell them to wash their hands." Well, everybody knew they were supposed to wash their hands. So we came up with another action item, basically preventing bare hand contact with ready-to-eat foods. And so from there, it gave that one additional level of protection, and as the manager of the facility, you could walk around and you could see your employee has gloves on. So hopefully, they're not sick. Hopefully, they've been washing their hands. But at least they have that one added layer of protection.

So from that, as you can see, this is basically a metric that shows the number of outbreaks that involved food workers for the period of 1980 to 2008, all right? So you can see a peak right around in the '80s, and then it started trailing off a bit to the -- you can see, right around 2008. You know, we still have outbreaks. Well, how come? Well, we still have compliance issues. Some people don't want to wear gloves, or some people work with food with their hands. So that's why we do routine inspections. Our field staff are trained. We ask them to cite it as a violation, take enforcement actions as appropriate.

So that's one metric. Another one deals with shellfish. Back in the early '80s, we had outbreak after outbreak related to shellfish. So if we tried to take the control action at the restaurant level or the store level, well, good luck -- they're already contaminated. It's not going to help. The intervention had to be back at the source. So – this is an oversimplification, but there were countless days and years spent setting up a regulatory program to ensure that the shellfish beds were approved for shellfish harvesting. In addition, tagging requirements were implemented. So all these different control measures were put in place.

So if you look at the metric for that, you can see that in the early '80s, there's a big peak, and then now it tails off for shellfish. And some of the ones that we're looking at now are actually related to Vibrio and not necessarily gastrointestinal illness from, like, a virus, or back in the early '80s we were dealing with hepatitis A issues, too. So an example of the metric for that.

And one more for today. You know, we were talking about shell eggs and Salmonella Enteritidis. We had a number of outbreaks that occurred, and it's easy stepping back now after all these years, because we know that the eggs themselves could contain the Salmonella Enteritidis. Back when I first started, as long as the eggs were intact, everything there was good. If there was an outbreak, the person did something wrong. It wasn't the egg, all right? So we had to get through barriers to show that they were actually contaminated at the farm level, and then if you consumed it undercooked or raw, like a number of people do, then you're going to be sick. And so we had a lot of scrambled-egg outbreaks because, people like them a little runny. And so, outbreak after outbreak.

So with understanding of at what point the contamination occurred, you can then set up multiple barriers. One, obviously, was back at the farm -- make sure they're safe when they come off the farm. And then through transportation, refrigeration issues. So we made a number of changes over the years, working with FDA, the CDC, to try to set up some different barriers to make sure that everything was done properly before it got to the restaurant, and when it did get there, the restaurant had a responsibility now to cook it properly and not serve raw eggs unless somebody specifically asked for it -- but very specific program requirements to address the issue.

So the metric for this is specific to Salmonella Enteritidis. Because if we looked at all Salmonellas, we wouldn't know if we're hitting our target or not. So you can see, in the green, it shows the egg-associated outbreaks. And it started low in the '80s and peaked up through late '80s, early '90s, and then back down. The other, in white, are the non-egg associated. So we could see the impact that it had, at least on our surveillance system. We have data right up to 2008 -- actually 2009, but we're just finishing that up. But it shows that similar trend. Periodically we have issues, but for the most part, we're not seeing those large outbreaks. Okay. Thank you very much. (Applause)MR. GUZEWICH: Thank you, Mike.

Our next speaker is going to be talking about the industry perspective to metrics. He's Dr. Bob Brackett. He's a Senior Vice President and Chief Science and Regulatory Officer at the Grocery Manufacturers' Association, often known as GMA.

Dr. Brackett oversees the association's scientific and regulatory activity, including the operation of its in-house food safety laboratory. Previously, he was Director of FDA's Center for Food Safety and Applied Nutrition, having first served in other positions in the agency.

Dr. Brackett also has been a professor of food science and technology in the Center for Food Safety at the University of Georgia, where he conducted research on food microbiology and safety. Prior to that, he was the extension food safety specialist and assistant professor at the University of North Carolina State -- NC State. Dr. Brackett. DR. BRACKETT: Thank you, Jack.

Well, Jack asked me to talk about the whole issue of metrics from the industry perspective, and so, as we've seen throughout the day, you've seen the same themes repeatedly by different sorts of perspectives. And so what I'm going to do is talk about some of the same things you have heard, but from the industry perspective, and perhaps give you sort of a different viewpoint of how metrics can be used to affect food safety in a positive way, that is, to help.

Well, I can give you the sort of bottom line right off the bat of what my perspective is. And that is that metrics are very important to the food industry. They drive their business model; they drive what they do in the plants; they drive what they are always striving to get the most activity, the most benefit, from every effort that they do.

But they question, may be more to the question, "but why would, in this context, food safety metrics be important to the food industry?" Well, there are a couple of different reasons, really, one of which is that it has an impact on government. And the first thing people would say about your industry, "Why do you care what impact it has on the government?" Because whatever impacts the government, impacts the industry, ultimately, in terms of regulatory policy that might be established; in terms of the public health priorities that are chosen that the industry must abide by; or even consumer confidence in government activities. It is ultimately essential that consumers have confidence in their regulatory agencies and in the food safety system in the country. And so obviously, it's important to the industry as well. But industry has its own interest in metrics, as well. It can affect risk and safety assessments that companies can do. It can affect the development of appropriate preventative controls that might be applied to reach government regulations or declared safety issues. It also can affect food safety plans, which will be much more important in the future when and if legislation is ever passed, including such things as plant design. If one is looking at trying to reduce the disease burden or trying to reduce the number of organisms in a plant and coming out in a food, such things as engineering come into play, and so that is something that one has to measure in order to be able to change. And, of course, industry does a certain amount of research. And having metrics to find out what works and what doesn't, in a quantifiable way, of course, is going to be quite important for research, as well.

Well, when we talk about industry metrics, really what are we talking about? First of all, it is not the goal of food industry metrics to achieve some public health goal at this time. But there are several different areas of metrics that are used in the industry that do affect food safety. There are, first of all, process records which are measurable data, such as time-temperature records of a retort, or used during pasteurization. Even though these are not considered to be metrics, they are still showing that the process is in control. Preventive control validation data -- if one company has cooking instructions on the package, have they actually validated to find out that those cooking instructions will actually eliminate the organism? That is a point where you end up measuring to prove to regulatory agencies that those cooking instructions actually do what they are intended to do. Such things as pH records, to make sure that acidified foods are, in fact, acidified.

There are many other types of examples that we could use here also. But this just shows you that these are routine records that are kept by most of the food industry.

In addition, there are what I call compliance data. This includes such things as audit reports, where one would have a score on the audit. It also may include surveillance data, and by surveillance data, I mean environmental sampling that is done in the plant or perhaps even end-product testing of the product, where one has some measurable result. These show that one is complying with what the regulatory requirements are, and I'll get to that point in a moment.

And then other sorts of things, like industry practices. And the example I put here happens to be mock recall efficiency, where one is aiming to measure how fast can I get a product back? How efficient am I at getting the product back? Do I have traceability systems in place that allow me to get a potentially harmful product off the shelf as quickly as possible?

So all of these are routinely done by much of the food industry. But again, as I said, the purpose is not necessarily the same purpose as CDC or FDA or FSIS would use in terms of applying the metrics.

So the question we have to ask, if that's the case, do these metrics actually result in safer foods, since that's what the goal is? And the answer is, not necessarily. Because many of these metrics are done for quality reasons; they're done because they are trying to meet regulatory requirements; and they tend to be rather much of a shotgun approach towards collecting data, some of which is useful to the agencies and to them, some of which is not. So with that being the case, what are we looking at? You have to ask the questions whether these metrics are actually even reflected in CDC's or the state's surveillance data. Some of them no doubt are, but there's really no way to make a one-to-one comparison the way things are right now. And if you follow that line of reasoning a little bit further, you see that we don't know whether these practices, these metrics that are being used, actually affect public health in a positive way, and that is the goal of actually reducing morbidity and mortality associated with foods. So the viewpoint that I'm proposing today is that perhaps instead of having the surveillance data or the public health goals be a reflection of what the industry is doing, to actually put the end at the beginning and say, perhaps the public health goals ought to be driving what metrics the industry is using, so that the proper metrics and the proper values for those metrics can actually be used. So really, we're talking about, you know, what is the right target that industry would use? Since they're already doing data collection, wouldn't it be nice if they could actually collect data that actually is helping meet some public health objective?

So at this point, I'm going to introduce, or to some of you re-introduce a concept that really aims at this, and that's what's known as the food safety objective, or I put sort of as a subtitle, "how safe is safe?" There is no such thing as 100 percent safety, but you can get to a point where it meets everybody's comfort level, in one way or another.

As was mentioned a number of times today, we have something similar to that now in the Healthy People initiative. That is not meant to be a food safety objective, but it is a goal. And it is a stated goal that the government agencies can work towards, and it has been exact targets that are provided. However, it's meant to be just that, a goal, and surveillance is often aimed at trying to see if we're achieving those goals, not for the opposite reason of actually driving the metrics that would achieve those goals.

So just to give you a definition, there is actually a formal definition for this, and I'll get to where you can see that a little bit later. But it is really the maximum frequency or concentration of a hazard at the time of consumption to protect consumers. And so this is not dissimilar to what Dr. Dreyling mentioned a little bit earlier in terms of having some sort of prevalence data that can show that you are driving toward meeting a food safety goal.

Now, the one question, and the most difficult question in my mind, is how do you come up with that one value or range of values that is going to be agreeable to protect the public health? And obviously that is a decision that is a national policy decision, or in some cases state policy decision, that requires active participation from consumer groups, from the industry, from policy makers, as well as regulatory agencies, to work together to come to something that actually does improve public health, rather than just saying, "Make this food as safe as you can be." That's sort of like saying, "Drive as slowly as you can without affecting the safety of the car." No, actually, speed limits are put on for that very reason, even though they may not be necessarily the safest way, which is to not get in the car at all.

Just to give you an idea of how this works, here we're talking about establishing performance objectives for the food industry, and there was a question this morning about performance measures. This is really what we're talking about. Performance measures that can actually demonstrate that at different points within the food processing chain, that you are controlling the hazard or reducing the hazard, and from those objectives you can actually develop control measures. That is, processes, interventions, controls, preventative controls that can actually aim to either protect the food or at least reduce the risk.

So here's a short example of how this would work. This is sort of a generalized food chain. You've got production, manufacturing, transportation, preparation, and consumption, and under this sort of model, what you would do is you would set these performance measures, those performance objectives, in between each stage. And what you're doing really is working backwards, as I said, as Dr. Dreyling mentioned. If you have the food safety objective at the point of consumption, you're asking the question, "What would it take before preparation to make sure that no one exceeds that, so that you can have the safest food?" And what would it take before transportation, or what would you have to do during transportation to make sure that that level is met? And what would you have to do during the manufacturing or production facility to make it to that? In some cases, these are processes that would eliminate an organism. In other cases, this would keep them from growing. But in any case, what you can do is then set up control measures or interventions that aim to get you to that point.

And then sort of the bottom line here is that all of these are measures, and all of these are metrics that could be used to help drive toward a food safety policy in the country that everybody could meet.

If you're interested in more about this, and I know many of you know about this, there is a very good document that's put together by the International Commission for Microbiological Specifications for Foods that's available on their web site. And if you can't read that, that's www.icmsf.iit.edu and so you can find out a little bit more. What I've proposed to you here or outlined here is a very, very simplified version of this, but I think it -- I would advise you to go and look at this and see whether you think that this would work in other areas.

So in summary, food safety metrics is of great interest and importance to the food industry and it always will be. However, food safety metrics should be used to reduce foodborne illness, not just to achieve some regulatory mandate. Government, industry, and consumers all have a role to play in identifying what those specific, appropriate measures might be, and that is something that I think with an integrated food safety system moving forward, that we would want to work in the food industry, with the rest of the consumer groups as well as the regulatory agencies. And with that, thank you. (Applause)MR. GUZEWICH: Thank you, Bob.

Our final speaker on this panel is Barbara Kowalcyk. She's going to be speaking from the consumer's perspective on food safety metrics. Barbara Kowalcyk is the Director of Food Safety at the Center for Foodborne Illness Research and Prevention, which she co-founded. Ms. Kowalcyk became involved in foodborne illness prevention in 2001, after the death of her 2-1/2 year old son, Kevin, from complications due to an E. coli O157:H7 infection. Ms. Kowalcyk has volunteered and spoken extensively as a consumer advocate for food safety. She has served on USDA's National Advisory Committee on microbiological criteria for foods, and currently serves on the advisory board for Georgetown University's Health Policy Institute's Produce Safety project and on two National Academy of Sciences committees. She is an experienced biostatistician in clinical research and the pharmaceutical industry.

Her family's experience with foodborne illness is a profound and moving reminder of why we are all here, working to make the food supply safer. Barbara Kowalcyk.

MS. KOWALCYK: Good afternoon. I was asked to come here today and give you a consumer perspective on the current status and future directions of measuring progress on food safety. Since I only have about 10 or 15 minutes to talk, I will be limiting my remarks mainly to data issues that need to be considered in developing metrics for measuring progress on food safety. I think you will find that my talk will reinforce many of the themes that we have heard here today.

In recent years, with the rash of highly publicized outbreaks, there has been a recognition that if we hope to further reduce disease and death from foodborne illness, we must move from our current reactive system to a more proactive system. This prevention vision, as some have called it, will require partnerships in the development of a food safety system that integrates science and risk analysis at all levels, to provide the greatest public health impact.

As Mike Taylor indicated this morning, this is a broadly shared vision of the approach needed to advance food safety efforts and prevent foodborne disease. In fact, with resources becoming scarcer and scarcer, it is becoming all the more apparent that we need a scientifically driven food oversight system that uses robust data to assess risks associated with food production and distribution, and then weight those risks to determine where limited resources would provide the highest level of food safety in order to protect public health.

The development of such a system is an enormous task and needs to be undertaken seriously, with due diligence. Now, I'm a statistician. Before I became involved in food safety, I spent about 10 years working in the pharmaceutical industry, doing clinical research. So I have a statistical approach to this problem. In my experience, statisticians are often brought in at the end of the process, not at the beginning, and are asked to make sense of what has already been collected. Maybe that's because many people don't really understand the role of statistics and statisticians. So I thought it might be useful to take a moment and talk about it.

Statistics is ultimately the science of data, not simply what you do after the data are collected. Statisticians are trained to tease out the critical questions that you are trying to answer, designing an experiment to answer those questions, and then quantify the uncertainty that most certainly will exist in the experiment.

In a way, statisticians are like architects. And this is one of my favorite quotes, by the way. "When you're building a house, the architect will ask you to describe your dream house, and talk about your lifestyle, tastes, and so forth to determine your housing needs. Invariably, you will not be able to afford this dream house, but it gives you a place to start, a goal. To cut costs, you may have to decide to get rid of the top of the lined cherry kitchen cabinets and the whirlpool tub. But you will eventually get to the point that the architect will say to you, "You can't cut anything else without jeopardizing the foundation and structural integrity of your house." What you can do is make your house smaller and expand later. You will certainly be advised to consider those future expansions as you build what you can afford now. Put in some extra load-bearing walls and the bigger electric box. Think about the flow of the house and so forth. It will save you the heartache and expense of having to tear it down and start all over again later."

So I think you've seen something similar to this diagram earlier today. This is our dream house, a proactive, risk-based food safety system that integrates epidemiology, risk assessment, and economics. Epidemiology is used to evaluate the burden of foodborne illness, identify major risk factors, and monitor trends over time, all of which is used to identify potential risk management strategies.

Risk assessment then uses that information to predict what happens when the system is changed and to select risk management strategies. Epidemiologic data can then be used to validate the results of the risk assessment, evaluate the effectiveness of the risk management strategies, and identify new areas of concern. And so the cycle begins again, much like the “plan, do, check” cycle of continuous process improvement. Of course, regulatory decisions must consider both the effectiveness and cost of an intervention. So the system works within an economic structure which is represented by the outer circle.

Now, the heart of this dream house is data, and specifically the attribution data which you heard about earlier today. Attribution data will improve our understanding of the causes and impact of foodborne illness, including the long-term burden. And once we understand that burden, burden costs and causes of foodborne disease, we can begin to establish priorities and determine potential prevention and control interventions. We must then evaluate each intervention or policy to determine its ability to positively impact public health at a reasonable cost and in a fair manner.

After we have identified our prevention and control strategies, we must define our targets, such as food safety objectives and microbiological criteria, against which we will measure our success.

Finally, we must measure the effectiveness of our efforts by comparing human health and food supply data against our defined targets. As stated in the 2009 NAS letter report to FSIS on attribution, "Developing foodborne illness attribution estimates will require a comprehensive program that combines different attribution methods and integrates many different types and sources of data."

Developing effective foodborne illness attribution models will require the systematic collection, synthesis, and analysis of data that relates food to foodborne disease. To achieve that, we will need to develop an integrated data collection system that tracks human disease, animal disease, and probably plant disease, too; food contamination, industry practices, behavioral patterns, and environmental exposures. Just as the integration of molecular subtyping in human disease surveillance has transformed public health, so will the integration of non-human surveillance with public health surveillance. Such an integrated data collection will enhance our ability to identify critical data gaps, and potential sources of foodborne illness will allow us to examine potential risk factors and influences and evaluate the effectiveness of prevention and control measures.

And finally, it will allow us to establish a link between the performance of food safety interventions and foodborne disease. In other words, it will give us the metrics for measuring food safety progress.

Therefore, it is critical that we develop comprehensive food, animal, plant, and environmental data collection systems that, when tied to public health data, can provide the metrics we need.

Now, we heard earlier today from Dr. Cuéllar that metrics are good for two things. First, they remind us of the progress we've made, and they remind us of what we still have to do. Without a doubt, we have made progress in reducing foodborne illness. But this progress has clearly stalled. It is clear that we are on the right track, but as this quote indicates, simply being on the right track is not sufficient. We have to continue moving down the track, and the question is how to do that in a way that allows us to meet our short-term needs while moving towards our long-term goals. Now, Dr. Bertoni said this morning, "You get what you measure," and I find this -- I really like this cartoon. I don't know if everyone can read it. In it -- I'm going to go through it for a minute -- Dilbert's boss comes in and says, "Do you have those budget numbers from last month?" "They're totally inaccurate." "I know, but those are the only numbers we have." "Actually, we have infinite accurate numbers to choose from." "Let's keep those in our back pocket in case we need them."

And Dilbert responds, "I'll encrypt them so no one else can use them."

Well, I know there are a lot of people out there who want to say, "We have to use the data we have. We don't have time to wait for perfect data." Well, I agree that we need to explore the data we currently have. We do need to keep in mind that the current data collection systems have evolved independently over time, which, coupled with a significant lack of data sharing, has resulted in highly fragmented data sets, creating significant data gaps that make it difficult to evaluate the effectiveness of food safety efforts. Ultimately, making do with the data we have is a reactive approach, not the proactive, preventive approach that we all say we need.

If we are going to go down this road, and we probably will, to a degree, in order to meet our short-term needs, we need to be clear about the limitations and uncertainties surrounding the data. Let's consider an example. Now, over the year -- let's look at FSIS' Verification Testing Program, which many of you probably know I've been critical of. And over the years, they have tried to make do with the data. And they've tried to use FSIS' Verification Testing Program data to estimate the prevalence of pathogens in meat and poultry products. However, as noted earlier, this program is strictly regulatory in nature and was designed to determine if a particular establishment was meeting a particular performance standard at a particular point in time. As far back as 2003, FSIS stated on its website that the data collected from this program was not statistically designed to estimate prevalence and should not be used to make year-to-year comparisons.

FSIS has made changes over time to the verification testing program to try and improve the representativeness of its samples, which I applaud, because representative samples is what allows you to use statistics to generalize results obtained through sampling to the entire population. Again, FSIS acknowledges on its website that these changes make year-to-year comparisons inappropriate. Yet some still persist in trying to use the verification testing data as an estimate of prevalence.

I think that there is a way FSIS could, and they absolutely should, design a testing program that meets the regulatory needs and provides prevalence estimates at the same time. This, of course, requires a proactive approach.

I can't tell you how pleased I am to hear today that FSIS is stepping back and looking at how they can improve their data collection system and move toward a proactive system of gathering the data they need to measure their progress. This is -- many of you will probably recognize this -- this is the “plan, do, check, act” cycle that underlies HACCP. It can easily be applied to data collection.

First, define the objectives. Second, develop the sampling plan. Third, carefully collect and verify the data, doing so using an integrated approach. Be transparent and, ultimately, share the data.

Now, I'd like to take a moment to talk about effective sampling plans, which are at the heart of any good metric. Robust sampling plans are designed to meet the objectives, address issues, such as bias, that could affect the representativeness of the samples, and take into account uncertainty in order to ensure the generalizability and interpretability of the data at the end of the day. In other words, you are trying to develop a plan that takes into account the key data quality indicators presented this morning by Dr. Bertoni.

I know some may view this as a statistician being a purist. But in reality, it is a scientifically defensible way to measure things in the real world.

Now, I know there are some who may think that I want perfect data. But as a statistician, I will tell you that perfect data only exists in textbooks. All data is dirty. When I worked in the pharmaceutical industry, I spent several years working on a new antipsychotic for schizophrenia. Trust me -- there is nothing perfect or clean about trying to measure if someone's hallucinations or paranoia have improved or gotten worse. Statistics is really all about recognizing the uncertainty, trying to figure out how to measure it, and what to do about it at the end of the day. Now, one thing we could do to improve the way we are measuring food safety progress now is to start talking about the uncertainty of the data we are collecting right now. Point estimates only tell part of the story, and many tatisticians find them to be of little relevance. The real story is in the uncertainty. It puts things into perspective.

Two common measures of uncertainty are confidence and power. The easiest way to understand confidence and power is to think about them in terms of our judicial system. I hope I'm not being overly simplistic, but I find that many people don't understand this basic concept. In our judicial system, we really don't want to send an innocent man to jail, and we don't really want to set a guilty person free. You want high confidence. That is, you want there to be a high probability that you will find an innocent person to not be guilty, and you want high power -- that is, the probability that you want to send a guilty person to jail.

One of the things that has been very encouraging to me today is the number of times that uncertainty has been acknowledged in the various presentations. Just a few years ago, it wasn't even being discussed in a public way. And that's important. Measures of uncertainty put metrics into perspective. They allow people to assess the precision and accuracy of the data being presented. Transparency about how the data was collected and analyzed, along with acknowledgment of the limitations, will provide the justification for removing the bureaucratic barriers to gathering and analyzing the statistical information needed to truly assess government actions and our progress in improving food safety. This is vital, given the emerging food safety challenges of the 21st century.

The agency should be applauded for today's meeting. It has gone a long way towards improving transparency.

I would like to conclude with a story. On January 28, 1986, NASA launched the space shuttle, The Challenger, despite concerns voiced by engineers about the air temperature. On that date, the air temperature was 36 degrees. No flight had ever been attempted below 51 degrees, and the manufacturer had insufficient data on how the O-ring boosters would perform at lower temperatures. Despite the concerns raised, NASA management decided to launch the shuttle anyway, and 76 seconds into the mission, the shuttle appeared to explode, and all seven crew members were killed.

As we all know, this was a tragedy, not just for the families involved, but for the space program, as well. Shuttle launches were grounded for over two years. While this is a dramatic example, it demonstrates the potential dangers of over-interpreting and over-generalizing data beyond the population sampled. As we move ahead in developing metrics for measuring the progress of food safety, it is important to understand that the public policies driven by those metrics can have profound impact on American lives. Our ultimate goal is to prevent foodborne illness and avoid tragedies like the Challenger and the ones happening to American families every day. Thank you. (Applause) MR. GUZEWICH: Thank you, Barbara.

We have time for questions. If you'd like to ask a question, please come to the center aisle, to one of the microphones here, and give your name and organization. And you can ask questions of anybody on the panel. MS. DEWAAL: It's Caroline Smith DeWaal at CSPI.

Erin, interesting presentation. I have one question and then one comment. I'm not certain that I clearly get the sense of what your attribution is for the FSIS-regulated portion of the burden of disease. So that's my questions.

I did note with interest that you said this is being coordinated together with CDC at the President's Food Safety Working Group. And building on Barb's presentation and her suggestion regarding the need for integrated data management, I'm wondering if the working group is considering the concept of combining risk assessment and data analysis, and even you could do some risk communication, similar to what's done in Europe with EFSA, the European Food Safety Authority? Thank you.

DR. DREYLING: Well, let me just talk about where FSIS is first with our attribution methodology. We have been working over the past two years, and even longer, really, to develop attribution methods that would let us really drill down to our specific products that we regulate. And that is the information that we need, to know how to target our resources effectively. And so along that process, we presented an attribution method publicly in January 2008, as part of a NACNE and public meeting. We took that approach to the National Academy of Sciences, and Barbara referred to being on a National Academy of Sciences panel. I believe she was on the standing committee, and a small panel from that committee reviewed our attribution methodology. We took their feedback back, and we have been refining our attribution method. And now, particularly because of the development of the Food Safety Working Group, we are working stronger than ever with FDA and CDC to coordinate our approaches, and we are having a series of meetings on attribution with these agencies and really trying to now take all of these different methods that we have developed and to come up with our strongest approach.

One of the things that our agency has been dealing with, and we're working collaboratively with CDC and FDA on this, how often do we need to get updated data, say from FoodNet, or other surveillance systems in order to do good attribution and illness estimates? And so that's one of the topics we're talking about.

In terms of data systems and integration, we are looking at how we can pull all of our available data together. There is also, through the Food Safety Working Group, an IT task force that is looking at data infrastructures across the different agencies.

MS. KOWALCYK: I just wanted to clarify one thing, and then ask Erin or someone from FSIS a question. First of all, while I do sit on those NAS panels, I represent today my own views, not those of the committee. And I did want to clarify, I was on the standing committee for that NAS -- I was on the standing committee that reviewed FSIS' use of public health data for attribution. I was not on the subcommittee and was not involved in the letter report that came out.

The question that I have for Erin or from anyone from FSIS is, to my knowledge there has been no formal response by the agency to that NAS letter report. And I was wondering if there was one -- if you had responded, and if not, when that would happen?

DR. DREYLING: Barbara, as we discussed, I know the small group of the consumer representatives did come in for a meeting with us, and we had a similar meeting with industry. We are working to revise the report that we took to the National Academy of Sciences, and we have a draft of that report that is going through management clearance to date. So we will plan to release that publicly in the coming month.

We also, in response to discussions that we had with the consumer representatives and other agency stakeholders, have developed what we call a strategic data analysis plan that really lays out how we plan to move forward in data analyses for risk-based approaches for the agency, for attribution methods. And so either with our decision criteria report or in a similar time frame, we will release that strategic data analysis plan, as well.

DR. GUZEWICH: One last question. Go ahead, identify yourself and who you work for? MR. McALOON: Todd McAloon with Cargill. A great meeting and great information. I'm just curious. Somebody said earlier in today's meeting that we need to work together, and I think there are thousands of people, both in the public and private sector, who want to produce safe food and reduce foodborne illness. I'm curious for some ideas on how maybe there can be even more active participation by the private sector and NGO sector in this Food Safety Working Group in order to have a deeper and more collaborative process, even on a day-to-day basis?

MR. GUZEWICH: I don't think anybody on the panel is part of the Food Safety Working Group. The President's Food Safety Working Group is a pretty high-level group. When we have our next panel, Malcolm Bertoni and Carol -- well, Carol's not on that one. But Malcolm is on the panel. He's been involved with some of those activities. I don't think anybody on this panel was part of that working group.

MR. McALOON: Okay. Maybe I misspoke. I'm talking about the collaborative process that you've all been describing today.

DR. DREYLING: Well, from FSIS' perspective, we do, as we are moving forward with developing metrics, try to engage our stakeholders from industry and consumer perspectives, and we do try to brief the public through public meetings and putting our reports public with public comments. But we are happy to engage and to hear, certainly, different perspectives about them. I can't comment about engaging in the Food Safety Working Group formally. MR. GUZEWICH: Okay. I want to thank the panel for their presentations, and we will reconvene at 3:45, which is about 12 minutes from now, for our question and answer period. (Short Break)

DR. MORGAN: All right. So thanks everyone for sticking around for the last session. This, of course, is a very important part of the meeting. As we all know, effective, successful communication is two-way, and so today you've had a lot of communication from us to you, and now is the chance for us to hear from you. My name is Kara Morgan. I'm the Senior Advisor for Risk Analysis in the Office of Regulatory Affairs at the Food and Drug Administration.

So you've heard from the agencies. You've heard from some other stakeholders in the food safety arena, and now it's your turn. It's time for us to hear from you. We've set aside this time to not only get your questions about earlier presentations, which of course are welcome, but also to get your thoughts and comments on what you've heard today. The agency folks here will especially be listening for ideas for future meetings.

As Mike Taylor talked about this morning, this is the first of a set of meetings we hope to hold. This is just the beginning of the conversation, and it's pretty clear, I think, from what we've talked about so far that nobody has all the answers yet. There are a lot of ideas that are in various stages, and then we're pretty confident there are lots of other ideas out there in the communities that are engaged in the food safety system in different ways.

So we hope to use these meetings to really have this conversation and to be working towards a system of metrics that will give us the information we need to make the decisions that we need to make. So the panel is here to respond to your questions, or in some cases redirect your questions, if they're from some earlier speakers, to the appropriate folks from their agencies.

And they're also here to continue discussion, if you have some kind of discussion points they might pick up on that and want to respond back and have that conversation. So I hope you'll engage in that. We've heard from all the panelists here except from Michael Batz, and so I'm going to, in a minute, have him come up and provide some comments from his perspective as a researcher, an academic in the world of food safety. But before I turn to Mike, I wanted to mention a few other things to kind of guide your questions and comments, hopefully for the session. So first of all, the questions at the end of your background, they were up on the screen this morning when Mike Taylor was talking -- and they're up there again.

These questions are what the FDA, FSIS, CDC group has been using to really focus our discussions about for this workshop, in particular, and we'll continue to be using those. So I hope that you can keep these in mind as you're developing your questions and comments, and maybe these will help trigger some specific ideas that you have, but this is really describing what we understand the struggle to be, and to help define what we're working toward.

And so also, as I mentioned earlier, this is just the beginning of the conversation. This is certainly not the last chance anyone will have to make comments. I'm sure there are a lot of folks out there with some very well-developed ideas, and we would like to hear about them, though, because of the short time we have today for public comment, I hope you can today try to keep your comments brief, kind of lay some ideas down, and that we can then pick up and maybe have a larger conversation at subsequent meetings. So please keep in mind other folks who might want to make some comments or ask questions during this time.

So now I will turn it over to Michael Batz. He is the Executive Director for the Food Safety Consortium and the head of the Food Safety Program of the University of Florida. He's also a researcher and head of the Food Safety programs at the university's Emerging Pathogen Institute at the University of Florida. His research centers on the use of information and analytic tools to improve regulatory and public health decision making, particularly in food safety. Previously he was based in the Department of Epidemiology and Preventive Medicine at the University of Maryland School of Medicine and was a research associate at Resources for the Future. MR. BATZ: Thanks, Kara, and thank you all for having me at this meeting today. This has just been a pretty fantastic introduction to sort of what the federal agencies are doing on the idea of metrics, and I think it's -- you know, I've been asked to give some remarks as somebody who is perhaps not constrained by the resources or constrictors of either the government or industry and I think today when I look at what's been presented, I think it shows a pretty remarkable move, say, over the past 10 years towards, you know, the sort of often-described science-based, risk-based, data-driven, evidence-based food safety system that we talk so much about as a goal.

In particular, I think we're hearing some of these presentations from CDC about focusing on attribution, and so many of these increased studies that have been done over the last five, six years are really remarkable and I just can't thank them enough for taking so much initiative in this area. I wanted to give a few words, stepping back a little bit from specific data programs, to think about what we mean when we say we want a science-based food safety system, and to me, recently, the more I think about this the more I think about science as -- well, it's been described -- I think Bertrand Russell said, “Science is what we know. Philosophy is what we don't,” and, you know, certainly I'm in favor of a philosophy-driven food safety system but, no, I think the – science starts with a hypothesis.

It starts with a question and the method of science involves an incredible amount of planning about where -- how we're going to address this question and some serious thinking about what kind of data we need to get there, what kind of analysis we're going to do to see whether that data answers the question, and some reporting out of the results. And so my thoughts about the role of science sort of relate back to that question of planning, and touch upon something a bit that -- you know, a little bit of what Barb said about sort of the process she described, and it's this idea that, as we move forward, I think we do need to recognize the difference between gathering data to answer specific questions that we have, versus making due with what data we have at hand.

And the challenge of our example is pretty dramatic. My own example is sort of Apollo 13. I think there's the situation in which you're stuck on the opposite side of the moon and you're running out of oxygen and you have to like, you know, develop some breathing apparatus out of duct tape and air conditioner parts and, you know, your tooth brush.

But I think we don't want our thinking about how we solve questions of metrics to be based on that kind of emergency, stuck on the side of the moon, kind of example. We want it to be planned. We don't want to, I don't know, “MacGyver” our way out of that -- out of whatever data we happen to have laying around. And that's not to suggest that we don't want to take advantage of data that exists.

Data is incredibly valuable and I think some of what we saw today really shows how valuable. We heard an awful lot of presentation about attribution based on outbreak data. But that compilation of outbreak data wasn't designed to address the question of attribution. It wasn't the model of taking these complex foods and assigning them to commodities. It wasn't part of how we – how those reporting forms were designed for states to report to CDC.

But I think some of the issues with outbreak data also show how more planning could be helpful. One of the reasons why there are so many complex foods in the outbreak data, and I say this as somebody who spent a lot of time trying to do very similar exercises in terms of analyzing it for attribution – the foods in there are not consistent or coded, or they're whatever the individual filling out that thing writes down.

So he writes down ethnic food was the vehicle. You might know what that means, but it's different to everybody. Hamburger might mean ground beef or it might mean a sandwich. And so when we -- if we thought that that data was going to be targeted for attribution, if we find that attribution to be of tremendous value of that data we should be thinking about how we can improve the applicability of that data attribution and maybe make it so that we're not talking about only 20 percent of known pathogens, known foods that we can sort of take advantage of. I think we also have to think a little bit about the applicability of that data to other situations.

I think it's a pretty good picture, overall, of where the burdens of disease are, but I think using it in regulatory settings becomes a little more challenging, because it reflects not the point of processing, but the point of consumption, which means that it reflects after preparation in restaurants or by caterers, that it's sort of the role of temperature abuse along distribution, sort of points that are subsequent to whatever point we happen to be regulating.

I think the case of chicken is an example where we know that a lot of cross contamination happens on cutting boards. And so if you think about it, if we know that outbreak data represents point of consumption, and we know that there's things happening during preparation, that's not just an uncertainty. It's a bias, because that cross contamination is one-directional. Nobody's contaminating their chicken because they sliced their tomato open, I don't think.

You know, what's happening is that the outbreak data, therefore, represents a picture that might -- depending on how you want to treat that cross contamination – over-attribute illnesses to produce, or under-attribute them to animal products. So for me, for attribution somewhere along that point -- that farm to fork continuum -- I think we do need to develop better point-of-processing models and point-of-processing data to support those models.

I think I would like to see further use of predictive microbiology and QMRA to sort of answer some of these questions. They don't have to take four or five years to do. They can be done much more rapidly and consistently, and used in this kind of environment. I think in terms of improving that point-of-processing, we did hear one talk today about the use of the CDC and FSIS collaborating on an attempt to apply the Danish model. And I just want to step back and say I think that that is such a great step in the right direction because, for years, people have been talking about the need to better integrate our human illness data collected by CDC primarily, and chiefly by states and localities.

The data that we have on animals, foods, feed -- I think that effort to connect those data is really critical and should be a model, but all of that said, I think it also reflects upon the need for improved data. We heard a little bit about the issues with shell eggs, but I think beyond that, if you look at the Danish Salmonella accounts, the intensity of data collection to support that analysis is -- is almost mind numbing.

They have prevalence rates that have been collected at multiple points in the farm, in processing for poultry, pork, beef and eggs, and they have that data so good that they can tell you prevalence of serotype and phage typing. So they have a prevalence level at this double T -- at this two tier level. Think about the depth of information that's in that kind of data set that's collected every year through a dedicated monitoring program that's designed to support that attribution analysis of that data collection is a huge part of why that model is successful, and I think that we need to do that. We need to invest a little bit more in those kinds of data if we really want to get good attribution out of those kinds of models. You know, I think that's a little bit easier said than done. Denmark is the size of Tennessee, and I don't know about you, but if I imagine Tim Jones with, you know, the resources of Denmark, I think it's pretty impressive.

Lastly I would like to note that all of that data that I just mentioned in Denmark collected is collected not actually by government, but by an industry trade association that manages that. So it's actually industry that collects that data. That's just sort of an interesting aside to me. To improve both kinds of data, human and animal data -- it was alluded to a little bit today, but I just wanted to reiterate just the absolutely critical role of states and localities in supporting our human illness data.

There's a front line; all the data that we see presented by CDC is part of our national infrastructure of 3,000 local agencies and all the different state agencies that work on that, and improving the quality of that data isn't as simple as sort of supporting programs at CDC.

We really need to think about capacity building in those states. A lot of states lack foodborne epidemiologists, and their budgets for food safety are just, in a lot of places, are in freefall. So I think that's a critical piece of improving the data that we want to support this. Lastly, I think what we saw today represents the last critical piece of science, which is reporting on results, and that sort of active dissemination is absolutely critical.

The more I see about these things and the more I think our agencies are producing interesting and incredibly valuable data -- the online outbreak data that CDC has made available for searching is just a phenomenal step in terms of making that data transparent and accessible. I would actually like to see a further step, and that's an integrated annual report that would combine data from numerous surveillance programs at CDC that would combine data from animals, food, and feed.

You know, there's tremendous value to sort of seeing this data side by side. It's often reported in many different stovepipe kind of places. You know, you want to find passive surveillance data. You find it over here, if the report has been published. The FoodNet report might be over here, or it might be in MMWR, with preliminary numbers or these things are -- as you're trying to take advantage of all these different data programs, some of the biggest advantage can just be putting them side by side so we can see what's going on with Salmonella data, what's going on with human data in the same report. Not in two different meetings sponsored by two different agencies, or whatever.

So I just want to put that idea forward as one that can really be valuable at integrating the data that we are collecting, and helping us to go back to that first step and prioritizing and planning what data we do need to collect to answer some of the gaps that have been shown through some of the other reporting. So that's it, and thank you.

DR. MORGAN: Thank you, Michael, and so again this is your time to ask questions and bring some discussion points forward, so please go to the microphones if you have some issues to raise. We have our questions here about how can we enhance human surveillance data programs, how can we enhance the attribution question. We've talked about that a lot today. What kind of metrics other than human disease data could be used to help tell the story about connections between interventions and between safety systems, and impact on illness, and are there existing data sources that just aren't well connected yet, as Michael was referring to, or are there other sorts of data that we need that you could talk about that we could consider in terms of moving forward. So do we have a question?

MR. KING: Hi. Hal King at Chick-fil-A. I have a question for Dr. Brackett, but I want to pose this in a recommendation. As I was listening to today's speakers about attribution of foodborne illness with a source of the illness and what were contributing factors to that illness -- and I'm coming from the perspective that I used to work at CDC in epidemic investigations, but now I work in industry and I think Dr. Brackett has a similar background in regulatory and academia and industry.

And as I sit and listen to his talk, I see a gap, and I just wanted to identify whether you thought this was a gap that you were talking about: Where we do look at the end point of what the disease rates are and what kind of disease people are getting? We do a really good job at making sure that we reduce the numbers of organisms in prepared food and the processing, but we don't -- I don't sense that we do a really good job of measuring how well we are doing interventions.

So the hallmark of public health science is good intervention. So a good measurement of that is that you have a disease, you have an intervention strategy, and then you measures of that intervention success, and that just drives that disease rate down. And we've got a lot of that built in for a lot of public health programs, like vaccination or diagnostics, to improve outcome of disease and things like that, but when I was watching Dr. Brackett's presentation, I started thinking we need to measure more the interventions that we have, and reinforce the regulatory on a lot of those interventions.

I know FDA is doing that at restaurants, where they're measuring what the risk factors are continuing to cause -- contributing to those kinds of things, but do we do it in the supply chain well enough? Do we do it in the processing, from the farm all the way through the transportation, to where it gets to retail? Thank you.

DR. BRACKETT: Was that a question or a comment? I was trying to -- well, yeah, I think that's right, and the whole point I was trying to make is that we do, and the term that I use sometimes is that the surveillance data that we have, public health surveillance, is really a reflection of what is going on in the industry, but it really doesn't show a cause or effect of the interventions that are going on in the industry.

Typically they will have the results of the intervention, but usually it's expressed in log reductions of microorganisms or something like that, but that doesn't necessarily transfer to fewer foodborne illnesses, because something else may be happening somewhere along in the chain that you're not taking into account. So I think what my purpose was that right now industry is doing that based on what they think is the regulatory requirement, as opposed to setting those interventions based on some target that perhaps public health or the regulatory agencies pose as being what is needed by the consumer, and then working backwards, because I'm convinced that if the industry knows what the target is, they can devise interventions that will meet those.

MS. BUCH: Hello. I'm Patricia Buch and I'm the Executive Director for the Center for Foodborne Illness, Research and Prevention, and not to pick on Dr. Brackett, but I have a question for him as well. In your presentation, I was very intrigued with the idea that you said that industry needed to set better public health goals to drive these metrics. Could you expound or expand a little bit on that comment?

DR. BRACKETT: Sure, Pat. Actually, what I said is that what industry needs to use is the public-health goals to drive interventions, and by that, it brings up a really good question. What are those values that are used? There's another concept that goes along with the food safety objective, which is called an appropriate level of protection, which some have interpreted as being how much sickness are we willing to accept, which is kind of a negative way of doing it, as opposed to re-posing the question as we now have 16 cases per 100,000 of Salmonella.

Is 10 an achievable goal that we can all drive to, and then moving it backwards? So there you're actually making an improvement using public health statistics to measure if -- and one would have to use something where one would have attribution, I think -- but to actually see if you can make a difference in the disease data, rather than just meeting a regulatory requirement to do something. But that is a question that, absolutely and societally, it would have to involve consumer groups and government and industry all talking about -- and coming to some sort of an agreement, much, I guess, like the model of dietary guidelines for Americans, perhaps.

MS. TUCKER-FOREMAN: Hi, I'm Carol Tucker-Foreman with Consumer Federation of America. Might as well keep on picking on Bob. Bob, I want to say that your discussion of the food safety objectives really brought me back full circle to an idea that was explored at length a number of years ago, and then it seems to kind of have been set on the shelf, and I was glad you brought it up. I think it bears more attention as we go forward here.

My question to the panel is, we all have a self-interest, or an interest in justifying whatever it is that we have decided to do. Regulatory agencies have a vested interest in continuing to do, continuing to follow the course of action that they've decided is the best one for accomplishing the goal of reducing foodborne illness. Where in the system is the disinterested, or if not disinterested, unbiased check on the data, if the regulatory agencies are the source of the data?

If the regulatory agencies are determining attribution, what is to keep the regulatory agency from following the course that makes their program look effective? I know that CDC is supposed to be looking at just the data out there but, you know, the truth is that that's also part of the government and part of an Administration, and if required to produce data on a much shorter turnaround than they have done in the past, that attribution data could be influenced by political consideration.

So where do we go, who do we refer this to for a final review that says we think that what you're doing in fact does reduce foodborne illness and yet get the most honest shot at review of the data that we possibly can. Now, in some countries they've divided the regulatory function from the risk assessment function and come a little closer to that than we have in our system. So what would you all suggest that we make sure we've got that unbiased? DR. BRACKETT: What I'm going to do is pass the microphone over to Barbara, because she and I have had this discussion about that very question about data.

MS. KOWALCYK: Well, I think you've raised obviously excellent points. You do need an unbiased source of data, and as you indicated, there are other models that we could look to do that. For example, the Netherlands, the data analysis arm of the -- looking at food safety sits in a pseudo-academic government agency called RIVM, the Public Health Institute for -- I can't remember the whole acronym but anyway, you know, and they really have some independence, or at least that's what I've been told.

And that might be a model that we would want to consider. You do need to have some mechanism for evaluating the bias and the data, which is one reason why I advocated in my talk today that we need to talk about uncertainty measures. We need to talk about confidence. One of the things that I was very happy to hear, as I just mentioned earlier, is that FSIS has talked about how they're trying to improve their verification testing program to better estimate prevalence, and they're going to -- obviously one of the ways mentioned was we're going to increase the sample size.

And my statistician hat immediately goes on and says that's great. So what's the power of the new sample size? That puts it into perspective for me, and that at least illuminates some of the biases that exist and gives other people outside of the government the ability to assess whether or not it's an appropriate use of data.

DR. BRADEN: This is Chris Braden, and one other thought that I had along those lines is that I'm not sure that we have taken advantage of Federal advisory committees, to the extent that we might, that have an outside and unbiased opinion potentially to provide. So that's one other thought that I've had, that at least CDC has not taken advantage of that to the extent possibly that it could.

MS. TUCKER-FOREMAN: If I could just respond to that. That's actually one of the things that advisory committees are set up to do, and I'm sorry, since you raised it, I refuse to leave the microphone without pointing out that the food advisory committee of the Food and Drug Administration last met in 2005.

I'm finishing a two-year term on that committee. It has never met, and the last time the national advisory committee on meat and poultry inspection met was August of 2008. I used to be a member of that committee as well. It hasn't met now for going on two years. That's a great idea. It's not quite enough, because I think you need an ongoing body, but one that can produce quickly. But certainly, when you're in a period where there is this much flux in how the government is dealing with a particular set of issues, there is need for better data, and utilizing those advisory committees would be a terrific idea. Thank you all.

MR. HOLMES: My name is Scott Holmes. I manage the Environmental Health Division with the Lincoln-Lancaster County Health Department, Lincoln, Nebraska. There are very few local folks here, so I'm going to try to speak a little bit from that perspective.

I currently serve on the council for improving foodborne outbreak response, which is sort of a collaborative group of basically most people we see sitting up there from the Federal agencies, and then I serve also as a local representative on the Partnership for Food Protection with the FDA and some other hats I wear.

I want to look at the third bullet on the slide, and specifically talk about what kind of metrics or other models outside of the human disease would enhance our ability to measure the progress on food safety. Mike Taylor, this morning, specifically brought up a couple of comments on that, as did Malcolm, and particularly Malcolm mentioned the percent of retail and food service establishment with adequate controls, proportion of consumers that follow food safety practices.

These issues are ones which are pretty much addressed at the local level right now. There are measures which are macro measures, you might call them, that apply to the local level. We can't measure at the local level and have a reasonable hope that it's going to show anything, something like the E. coli O157 incidence rate. It's ridiculous. If we have one outbreak, it will completely blow our rate for 10 years.

So it does us very little good to measure -- it does us very little good to know what's going on with our programs. FDA has created the regulatory retail food standards, which are a voluntary quality assurance system, that quite a few jurisdictions around the country have tried to utilize to improve outcomes at the local level.

Yet today I haven't heard anything about the FDA voluntary regulatory retail food program standards. Those standards are a very comprehensive quality assurance system which can result in measurements over time, but more on a macro scale is the percent of communities that are actually currently subscribing to such standards.

FDA's goals are to get about 15% of jurisdictions to participate in the standards. They have a goal of 15% of eligible jurisdictions in the standards by October 1 of 2010. They're pretty close to that goal, and have at least 50% of the enrolled jurisdictions meeting at least 25% of the standards. There are nine different standards. Those are extremely minimal goals.

If there were some real incentives involved for local health departments and state health departments to pursue those standards, there might be some progress. Every one of you ate some food somewhere, I trust for lunch, or had a snack today. I'll guarantee that FDA, USDA, CDC did not do the inspection in the restaurant of the people handling the food, but a local person did. And the reason I bring that up is that we're not talking anything about what's going on at the local level. One last aspect, Kevin Smith presented about the work that's been done to establish baseline data across the country on the retail compliance, in essence with the FDA Food Code. There are audits performed in many states of the local regulatory food programs that are quite similar. We have that outcome data. We know in our jurisdiction, for example, that our critical item violations have reduced to about 30% in the last four or five years.

Our overall percent of violations has dropped about 15%. If there is no study out there that's looking at factors such as things that we know -- we know that if you have a mandatory food manager program, like we do, that you have fewer critical item violations. We know if you have mandatory food manager certification, you have fewer outbreaks.

We know if you have fewer critical item violations you have fewer outbreaks. All these things interrelate, yet we're not looking at a community-by-community comparison to see if we're seeing differential outcomes in the other side, and that's the disease that we're measuring, such as through FoodNet. So there could be a very large study composed, something like the 10-city studies from the past, relative to heart disease, that could be done with the power, I trust, I hope, Barbara, that could identify which of these factors that we know something about on the retail side actually correlate with disease factors.

Finally, I guess I would like to mention that the retail food safety standards are something that are easily applied and are having, I think, some impact, yet there is no funding coming, really, of any large measure, from the Federal government to local retail food inspections. Throughout the country, with the budgetary reductions that have happened at the state and local levels, there are many food programs that have reduced staffing. There are many food programs, in fact, several food programs, that have been eliminated actually at the local retail level, and if there's not a change in that funding coming down to the local level, you will not see any improvements relative to these outbreaks.

DR. MORGAN: Thank you. Does anyone on the panel have any comments on that? Go ahead. Barbara? MS. KOWALCYK: I think that one thing we heard throughout the day, pretty much over and over again, is how much we rely on reporting of foodborne illness, and that really does happen at the state and local level. And without the cooperation of the state and local level, and I think Mike alluded to this earlier when he spoke, you know, that's really the heart of getting at some of the data that we need.

The trick is how do you standardize that, because we need to be able to aggregate that data and we need to also be able to drill down, and that's going to require a process for setting standards and having all the states collect the data in the same way. And you touched upon a very important issue, I think, and that's one that my organization and I think other consumer advocates are very interested in, and that's in funding. The state and local agencies are having significant funding issues, and let's take PulseNet, for example. I've been informed that the PulseNet team in Pennsylvania has basically been gutted because the states, due to budget problems, are not replacing people when they leave.

And PulseNet receives part of its funding from USDA, FDA, CDC and from the states. If we don't address the funding issues at the state level, how are we going to be able to support the surveillance systems that we have? You know, we need PulseNet to be working in each state. We need FoodNet to have the funding that it needs. So when we look at this overall picture, and there's been a lot of attention paid, I think, to FDA and FSIS, we need to also pay attention to these state and local surveillance systems that are in place, because they really are going to provide us with the information we need, and it's going to take strategic planning to be able to get where we need to go.

MR. BERTONI: Thanks, Barbara. I would just like to add a couple of comments from FDA. This is Malcolm Bertoni. I think it's important to acknowledge the challenges that the gentleman from Nebraska put out here, and there are a lot of issues that you raise that are excellent ones, and I think we all acknowledge that there are some new ways of thinking about things and new ways to look at the problem that we have to address. I would also say that, you know, for FDA, we've been trying to move more in a direction of doing a more coordinated and integrated approach to planning our work with state and local governments.

There's been a lot more movement in that direction, so that's definitely moving in the right direction, but there's clearly a lot more to be done. And I also think it's important to recognize that this meeting today is just the first in a series of public meetings and we are still planning the different topics, but I believe clearly, high on the list, is one that is going to address the state and local issues.

So even though today we spent a lot of time talking about some of the challenges, particularly methodological challenges, that were going on with the Federal programs, we do plan to be speaking more and having more engagement on the state and local issues.

We know it has to be a partnership. Collaboration is another theme we've heard today, and we really appreciate you raising those issues.

MR. McALOON: Hi, Todd McAloon with Cargill. We heard this morning that outbreak data and illness data reporting is voluntary, and I was just listening to the conversation, and I was just curious. Is the current system working? Is this voluntary system working, and if not, has there been any discussion about making it mandatory?

DR. BRADEN: So is it working? Well, I would say the intent is working. We actually have great relationships with our state and local partners, and there is always great participation when people come together to pursue a new idea or a new project, whether it be within FoodNet or among other states, or even starting a new kind of network of research in the evaluation sites that we're trying to do now, standardizing the epidemiologic side of data collection.

I don't think we need to go the route of mandatory reporting. I think that people are plenty willing, at the state and local levels, to share their data with CDC, and given certain data-use agreements. What doesn't work, I think, are some of the issues that have been raised before, having to do with the fact that a national program for food safety starts at the state and local level, and yet they're not getting the resources that are required to support a national food safety system.

There are a number of circumstances in which we ask states to do work that they wouldn't necessarily otherwise do, because we want them to do that as part of the national food safety effort. The easiest example would be when we have multistate outbreaks. There is a lot of work that goes into interviewing every case, but that wouldn't be done if a state has only one or two cases, unless we asked them to.

And so there is a lot of effort that goes in that we, at a national level, are asking state and local people to do that they wouldn't otherwise do, but it's an unfunded mandate. But it's not really -- it's not really a mandate. They're doing it voluntarily. They want to do it, but there's a scarcity of resources. So I think that's something that we really do need to address, and as the agencies, we need to work together to have, I think, a common plan to support state and local agencies and officials in the work that they would do to support a national food safety program. DR. MORGAN: Go ahead, Barb.

MS. KOWALCYK: I just wanted to follow up on that and, you know, I agree completely with Dr. Braden that the states do have an unfunded mandate. Although I would not underestimate the lack of data sharing or the data sharing problems that exist between the state and local agencies and the federal agencies. I've heard story after story where one local agency won't talk to the state health department and the state health department won't give the information the CDC needs, and CDC doesn't give information that the state and local people need.

And, you know, it all depends on who you're talking to. But there is -- to me, there is a clear misperception of what data can and should be shared across these agencies. And we all really have to step back and look that we're on the same team. We're all trying to answer the same question, and we need to share data more effectively, not to -- the unfunded mandate is certainly a big, big part of the problem, but we do need to, I think, educate all the parties involved on the importance of sharing data and how we can overcome, quite frankly, what are the bureaucratic hurdles to effectively doing that.

DR. GOLDMAN: One other quick follow-up, going, actually, back to Scott's comment a minute ago, appreciating that at, the local level, a large outbreak of anything can overwhelm your resources for some period of time. I think the effectiveness of voluntary reporting is most acutely seen when the regulatory agencies make a change in policy that has resulted from a careful investigation into an outbreak investigation, and I think we all probably need to do a better job, at the Federal level, of not only just taking in information that's provided from a local level on through the states and up to us, but transmitting that back.

Sometimes that cycle can be months or even, in some cases, years. But I certainly have two good examples of cases in which a series of outbreak investigations of a particular pathogen and a particular product have led our agency, over time, to make some policy changes that we believe are more protective of public health, and it really relies completely on the efforts of Scott Holmes and his department and the 3,000 other local health departments, because that's where every outbreak investigation starts.

So I want to exhort you to continue to do that, even though you can't see any immediate payoff, because it does help us.

DR. MORGAN: Are there any more questions or comments from the audience?

MS. SOLOMON: I'm Goody Solomon with the Food Nutrition Health News Service, and I've been hearing a lot about the inadequacies, deficiencies, the problems we have with data collection and the relationship to reduction of disease and foodborne outbreaks. Which of these would you say is the most serious, having the greatest ill effect on the public, if you want to put it in those terms? And also, the second question, of all the fixes we've been hearing, which one would you say is the most likely to occur in the shortest time?

DR. MORGAN: I'll give folks a minute to think amongst themselves. Thank you.

DR. BRACKETT: Yes, I thought part of the idea was to get the audience to answer questions, too. I can only sort of give you my opinion. I think, in my view, that the attribution question is probably the single most important thing. It's not going to be necessarily done the quickest. However, if you do have interventions or regulatory solutions, when you do have attribution, those are the things that are going to have the quickest payoff.

And a good example might be the implementation of the shell egg rule for Salmonella Enteritidis or if you had with Vibrios, if the shellfish industry were to adopt the guidance that was given and the results of the risk assessment, something as simple as very quick refrigeration of oysters on the boats, that is likely to have a big impact as well. So there are simple things you can do. You don't have to have giant risk assessments.

MS. KOWALCYK: I guess I'll just say I concur. I think that we all agree that attribution is really the heart of it. As Bob said, it's not going to be easy, and it's really -- it may sound simple to say, but it really is going to take a multifaceted approach in coming at the problem from many different angles. And we're going to have short-term goals and long-term goals. We need to set up some systematic way of collecting data other than human health data. We need to start looking at how well the interventions are working. We need to look at human, animal, plant and environmental contamination, and that will help drive attribution.

So, you know, it's hard to say just one thing that would improve the system and have the biggest public health impact, but I think that there are a lot of small things we could start doing, and quite frankly, this public meeting is one of them, and that's increasing transparency and a better understanding of what all the different parties are doing.

MR. BATZ: I think, specifically, in attribution, though, when we start parsing it down to across, say, the 31 pathogens, across 16 or 28 or 40 commodities, depending on how you break them down, some of the -- what some people like about the outbreak data is its ability to cover the landscape, but there are some pretty big gaps in that data.

You know, toxoplasmosis is, at least according to me, and I don't know where it is in Scallan or Daughter of Mead or whatever, but the – you know, Toxo is the third -- causes the third most foodborne deaths, according to Mead, and yet it's a pathogen that we really don't address very much. There's been recent work that suggests maybe we're overstating the degree to which cat litter is responsible, and I would argue that that's a serious hole.

I don't know too many parasitologists, but I know that they're important. I think another pathogen for which attribution is pretty rough is Campylobacter. You have a lot of conflict between what outbreak data says, because Campylobacter doesn't occur there very much. And what experts say in an attribution that me and Sandy Hoffmann, who spoke earlier -- which she did and which I [indiscernible] and case/control studies done by FoodNet, they tell different stories.

Campylobacter remains a hugely important pathogen at the top of the list, and to understand the relative role of poultry versus, waterborne or more direct contact is pretty critical. And I think that means, to me, what I think is fairly well represented in outbreak data, but which we know is incredibly complicated, as Salmonella, with the serotype distributions, and there have been issues for years about connecting these things to food products.

I don't know that that's a short-term fix, but I think Salmonella attribution is going to remain probably the most important one of them. And I don't really have a big fix, other than to throw my hat in the ring again for increased reporting and unified reporting. I think nothing would do more to help consumers and those of us who care about food safety, to try to see more and more information gathered in a single spot, so we don't have to go look for the notifiable disease data on that section of the website, and then the FoodNet report over there and then just try to put that together and see what baseline study data are available in Excel.

And it's great that the different agencies and offices are publishing this data, and, I mean, I'm super thankful for that. I can't even express that enough, but I would like to see increasingly, as we move forward toward something more integrated, where those things are reported together.

DR. MORGAN: I think we have time for one more question. MS. GROOTERS: Hi, my name is Susan Grooters. I'm with STOP, Safe Tables Our Priority. To sort of piggyback on something that Mike is saying, and some things that we're hearing about, data gaps and Campylobacter, to my knowledge Campylobacter still isn't a nationally notifiable disease. So we're relying on FoodNet data and some PulseNet data to sort of give us any kind of idea about Campylobacter.

Maybe it's time to consider making Campylobacter a nationally notifiable disease. I don't know if anybody wants to say if there's any discussion about that, or if there is any movement on that, or how to get it to be a nationally notifiable disease?

DR. BRADEN: I would have to actually check the history to see if Campylobacter has been brought up as a nationally notifiable disease to the Council for State and Territorial Epidemiologists before. I don't recall that that's the case. In any event, that would be the body that is the partnership between CDC and the state and territorial epidemiologists that would consider, you know, adding as a nationally notifiable condition Campylobacter or another one.

Campylobacter is hard, I think. It's hard for a number of reasons. Number one is that it has probably quite a wide variety of vehicles that it's transmitted through. Also, when we look at it, we've cracked some of these nuts by subtyping, and Campylobacter doesn't fall out very well when you subtype it and you try and then look at clusters of Campylobacter.

The whole methodology of genotyping and subtyping for Campylobacter doesn't work as well as we have seen for Salmonella and for E. coli O157, for instance. You know, I think what we need to do for Campylobacter is try to learn more about its epidemiology so we learn more about what we can do with it. If we knew more about what we could do with Campylobacter, people would be more willing to count it as a nationally notifiable disease.

It's one of those that kind of falls through the cracks as far as that's concerned, because it is a difficult organism to understand well.

DR. MORGAN: Any other comments? MR. BATZ: Yes, I just wanted to note that - I think the subtyping issue is definitely there, but New Zealand just recently published an application of the Tine Hald's model using MLST on Campylobacter, and it's pretty interesting. They made some improvements to the model, for all of the model nerds out there, and it's pretty interesting. So I think Campylobacter and FSA just published some guidance on source attribution for Campy, so I think there may be some opportunities there to learn from Europe who -- the Netherlands have been focusing on Campylobacter for the last few years and there may be some opportunities there where we can learn to what they're doing, from what they're doing. Sorry.

DR. BRADEN: Yes, I agree. It's a great area of study, and we need to concentrate on that, but it's behind a lot of the other organisms, as far as what we know about it.

DR. MORGAN: Okay. We're going to move on to closing remarks. Malcolm Bertoni is here on behalf of Mike Taylor to close the meeting.

MR. BERTONI: Well, first of all, let me thank everyone for spending the day with us. It's been really a great day from our perspective, and it's been great to have the participation with the audience. As I mentioned before, really, this is just the first in a series of public meetings, so we hope that you will stay engaged as we plan and look forward. Also, just wanted to be clear about the status of all the presentations that you saw.

We are going to get those onto the FDA's website, where we have, I guess, the same location where this meeting announcement was. We hope to have that by Thursday. If you made any specific requests for slides, I'm sure the speakers will comply with that request separately. So all those interesting graphs and tables that you couldn't read from your seat you'll be able to look at.

I wanted to offer a few personal observations. My synthesis made some common themes that we heard today, and leave a little food for thought, perhaps. First of all, I think it's clear that we're making progress, and yet there is much work to be done. It's very clear, when you look at the Scallan work and some really important advancements, like looking at uncertainty, that's really a key issue, and to have that more explicitly addressed and transparently addressed I think is particularly important.

You know, it was about 20 years ago I was in graduate school in Foggy Bottom in GW, and was studying some of these things like Bayesian methods and Markov Chains and Monte Carlo simulation, so to hear Dr. Cole talk about some of these things kind of warmed my heart. And I think it's particularly important that we look at this, because it is complex. We need to educate the public about the fact that science is not always cut and dried, and to be able to help them understand where we have consensus and where we need more information I think is critically important.

Knowing that we have all this work to do, and knowing that we can't do everything at once, particularly given the resource constraints we all have at any given time, I think it's important that we make our choices strategically about what we're going to take on. Now that, of course, is a challenging thing, because all these problems seem very important. So we are going to have to use public health risk as an important criterion for making those choices, and yet I think we can't do that alone.

That's why events like this are very important to get feedback, and there are other forums where we can get feedback. In essence, I believe what we're building here is that we're building a learning system. We're learning about what's happening in the food chain, what interventions work better, and I think that, to do that, we're going to have to look at a number of ways where we can shorten the feedback cycle, more timeliness of data, and there are technological aspects to that; new technologies are coming online all the time.

I'm sure many of you in this room are carrying around some kind of wireless device that has more computing power than was in an entire room perhaps 30 years ago. And so things are changing, and our ability to bring data together is increasing, and I think it’s going to change more rapidly as we go forward.

The other thing I think is pretty clear is that collaboration and participation are very, very important. You've seen evidence today of the collaboration that's happening at the federal level. You've heard a little bit of talk of collaboration at both Federal, state, and local levels, and there needs to be a lot more of that. I know there was one very challenging question we got there: what can you do quickest.

And I guess, in some sense, where you stand depends on where you sit. And I run a planning office, and to me one of the best things we can do is more of this, better planning, better sharing of information, and connecting the dots through that kind of transparency and collaboration. I think, though, in the end, we really need to remember why we're doing all of this.

It's, yes, to improve how we're regulating and improving public health, but ultimately it does get down to every one of us who has to eat, and protecting the safety of the food supply, and also making sure that there is public confidence, consumer confidence in the food supply and in how well we're overseeing it. I think that was an excellent point that Dr. Brackett brought into this, but we can't lose sight of that, because if the public loses confidence in the regulator's ability, or other public health agencies’ ability to protect it, if we lose confidence in the ability of industry to produce, through their supply chain, safe and nutritious and wholesome food, then we all lose as a society and in many different ways. So it's important, through this measurement system, and through the transparency and participation that we're doing through forums like this, I believe, to help strengthen and build up that trust and confidence in what we're doing.

Really, I don't want to spend a lot more time making observations. I think it's important for us to take stock of what we heard today, look forward to other topics that we might be able to present in these kinds of forums, and work together to make it a better food safety system. Thank you again for your participation and your attention, and we look forward to other meetings coming up soon.