As a continuation of my IBM/Partnership for Public Service Podcast series of discussions about using big data and analytics in a number of creative and interesting ways to support real-world business goals, Conversations on Big Data, we recently discussed with Malcolm Bertoni, Assistant Commissioner for Planning at the Food and Drug Administration, the important elements of a successful analytics program and a road map for getting there.
By way of background, in the late 1980’s and early 1990’s, the Food and Drug Administration faced a mountain of criticism. It was thought that the public health safety precautions built into its drug evaluation procedures in reaction to the Thalidomide tragedy two decades earlier were responsible for delaying consumers’ access to vital new drug therapies. Particularly in light of the growing activism around fighting AIDS, critics argued that the FDA procedures were born out of disaster and therefore extremely overcautious. In response, Congress passed The Prescription Drug User Fee Act of 1992 (PDUFA) which enables FDA to charge user fees to drug companies in order to fund the work required to improve the agency’s evaluation process performance. The legislation also required that the FDA report to Congress regularly the agency’s progress in meeting the goals.
According to an article by Doug Schoen published 5/4/2012 on FORBES.com:
“The PDUFA user fee program has been a great success over the past two decades, as the increased funding from the pharmaceutical companies has allowed the FDA to . . . reduce the drug review process by nearly half. A 2002 Government Accountability Office report found that the average approval time dropped from 27 months to 14 months over the first eight years of the act, and user fees increased new drug reviewers by 77 percent.”
Malcolm Bertoni says that success came about because FDA achieved “the whole culture of managing to particular goals that the managers really care about”. Organizational and cultural changes are always challenges. Here are some observations Malcolm shared with Laura Shane and me recently regarding how FDA accomplished it.
At the beginning, there was some push back when we engaged various program areas to collaborate on how to improve performance. Fortunately, there was very clear Executive Sponsorship; the law requires regular reports to Congress and the President that were scrutinized very closely by activists at the beginning. That need for transparency, as well as the fiduciary aspect of the user fee structure also imposed a mutual responsibility between the FDA and pharmaceutical companies that fostered collaboration from a business perspective, which resulted in the early establishment of Specific, Clearly Defined Goals.
FDA then built on cross-agency Partnership by building on the similarities between drug and medical device trials to performance management. Both sets of processes involve collecting huge volumes of data and evaluating that data against specified criteria. The definition of metrics, whether that metric is the number of successes within a drug trial cohort or the numbers of drugs approved per year, is a scientific process with which FDA staff are very familiar. In fact, there were many performance management processes already in place within the various agencies, of which staff were somewhat protective. Malcolm says that the FDA made sure to emphasize the real goal of instituting performance management – ensuring that the American public has the fastest possible access to advances in medical science and treatment of disease – and to emphasize that it is NOT in any way punitive. The scientific methodologies already in place at FDA helped internal stakeholders see the value of standardization and to create sets of well-defined processes that both enable better delivery of the mission and as ever improving performance management.
Once the stakeholders were all engaged and some initial standards and processes defined, the process of defining Key Performance Indicators began. Malcolm says that there needs to be a combination of Top Down and Bottom Up contributors to develop a strategy for performance improvement. The measures needed to be both efficient and relevant. Efficient measures are clearly supported by data that can be captured and analyzed relatively easily. Relevant measures are easily understood in the context of the process they are applied to. According to Malcolm, without the involvement of the process actors at the agency level, the importance of some measures can be overlooked. By discovering the relationships between the processes themselves and performance measures, a set of KPIs are developed that are:
· Clearly Defined
· Aligned with Strategic Goals
· Easy to Measure
· Relevant to the Process
· Attainable
According to Malcolm, once it became clear that the performance management initiative was more about defining goals than assigning blame the FDA staff “discovered the power of actually measuring and tracking something . . . all of a sudden, they get an ‘A HA!’ and they can see . . . the value of applying the same scientific principles used in application review or risk analysis to actually managing the program and the resources.” That culture shift from silos of ownership to collaboration across organizational lines with real acceptance of performance goals as part of the public health mission is the main enabler for FDA’s success in using analytics to measure and improve performance.
As proof of this success, Malcolm cites FDA-TRACK. “I invite people to go to the FDA website and search for FDA-TRACK . . . because it’s kind of all of the above in the sense that it is a visible external portal into FDA performance . . . what FDA does and how we do it”. Malcolm describes FDA-TRACK as a ‘regular conversation about performance’ as well as a dashboard where one can drill down through performance data. Those conversations are needed to ensure that the dashboards are relevant to the areas addressed as well as to the strategic goals of the organization.
For the complete full audio interview, please visit http://www.ourpublicservice.org/bigdata/.
Leave a Reply
You must be logged in to post a comment.