Biostatistics & Biostatisticians in the Policy Arena

Transcription

1 Biostatistics & Biostatisticians in the Policy Arena Thomas A. Louis, PhD Department of Biostatistics Johns Hopkins Bloomberg School of Public Health tlouis/ ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 1

2 Outline Roles and Goals The Policy Context Quantifying and Communicating Uncertainty Politics & Statistics Successes, failures and threats Education for Policy Involvement Coda ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 2

3 Roles and Goals We need to insist on proper roles and other rules of engagement We need to bring high-level statistical thinking and technology to the policy context We need to respect and promote our discipline and ourselves We need to be statistically, scientifically and socially responsible, but not shy ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 3

4 Level 0 Design For a project to be successful (and valid!), it must include the right group of investigators, and be sufficiently resourced Just because the study addresses [fill in your favorite topic] does not imply that the study should be headed by a person in that field If the study is quantitative to any degree, at least it should be co-headed by a statistician or a non-statistician who is sufficiently numerate And, the team should be sufficiently broadly based ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 4

5 Level 0 Design For a project to be successful (and valid!), it must include the right group of investigators, and be sufficiently resourced Just because the study addresses [fill in your favorite topic] does not imply that the study should be headed by a person in that field If the study is quantitative to any degree, at least it should be co-headed by a statistician or a non-statistician who is sufficiently numerate And, the team should be sufficiently broadly based Nothing is purely statistical, but essentially everything has statistical content ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 5

6 Two examples The Navy s study of ecologic effects of Very Low Frequency Radio Waves The study was headed by an electrical engineer and no statistician was on the core study team It was poorly designed, conducted, analyzed ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 6

7 Two examples The Navy s study of ecologic effects of Very Low Frequency Radio Waves The study was headed by an electrical engineer and no statistician was on the core study team It was poorly designed, conducted, analyzed Keeping this perfect pitch, no statistician served on the NAS panel that reviewed the study! ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 7

8 Two examples The Navy s study of ecologic effects of Very Low Frequency Radio Waves The study was headed by an electrical engineer and no statistician was on the core study team It was poorly designed, conducted, analyzed Keeping this perfect pitch, no statistician served on the NAS panel that reviewed the study! Contention wrt clinical trials of t-pa s effectiveness The NINDS asked three statisticians to review the studies We insisted that the team be enlarged to include three clinicians, one who was quite critical of the conclusions drawn from the studies Our report supported study conclusions, and its credibility depended on clinical sophistication and study team composition ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 8

9 Two examples The Navy s study of ecologic effects of Very Low Frequency Radio Waves The study was headed by an electrical engineer and no statistician was on the core study team It was poorly designed, conducted, analyzed Keeping this perfect pitch, no statistician served on the NAS panel that reviewed the study! Contention wrt clinical trials of t-pa s effectiveness The NINDS asked three statisticians to review the studies We insisted that the team be enlarged to include three clinicians, one who was quite critical of the conclusions drawn from the studies Our report supported study conclusions, and its credibility depended on clinical sophistication and study team composition It also depended on statistical sophistication ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 9

10 Statistics in the policy context Generally, relatively weak empirical evidence Some from experiments; much from observational studies There can be a lot of data, but relatively little information (flat likelihood) So, priors and loss functions can dominate the decision-making process And, agreement on the likelihood (the model) is frequently contentious Biostatisticians operating in the policy context need to be aware that science, politics, economics, personalities, goals/motives/intentions are all in play And, that sometimes there is a value to uncertainty ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 10

11 Policy Domains are Permeated by Uncertainty Little direct information Considerable model-based uncertainty Even more model uncertainty Even if all stakeholders agree on the evidence (fat chance!), different conclusions are the rule (different loss functions and different priors) ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 11

12 Quantifying and Communicating Uncertainty We must communicate valid, credible and understandable information The most understandable and high-impact messages avoid communicating uncertainty But, science, policy and ethics require that we communicate relevant uncertainties And, possibly, incorporate uncertainty into the decision-making process A careful titration is necessary to produce a reasonably honest communication that has the appropriate impact, but also preserves long-term credibility Research Topic: How to do this ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 12

13 Model-based Uncertainty and Model Uncertainty Model based uncertainty assessment is easy But, when there are little or no data to inform about models (when the likelihood is not informative about model choice), prior uncertainties migrate directly to the posterior distribution Weights in a prior that mixes over models are changed very little a posteriori So, injecting lots of prior uncertainty, injects lots of posterior uncertainty as to model choice This uncertainty propagation operates in what if analyses and in Bayesian Model Averaging ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 13

14 A low-dose extrapolation example Linearity or other shape depends on the dose metameter Shape also depends on the fundamental relation between exposure and dose Many models fit the observed data well, but give radically different estimated safe doses Sometimes relatively ad hoc estimation approaches are used ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 14

15 Model Uncertainty: Liver Tumors in Rats consuming 2AAF Low dose model Virtually Safe Dose (10 6 elevation) Linear Multi-stage* Weibull* Gamma* Logit Probit * = model fits observed data ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 15

16 An air pollution and health example Koop, and Tole (2004). Measuring the health effects of air pollution: to what extent can we really say that people are dying from bad air? J. Environ Econ Mgmt, 47: Show that using Bayesian Model Averaging and a broad range of candidate models, there is a great deal of uncertainty about the magnitude of the air pollution effect Thomas, Jerrett, Kuenzli, Louis, Dominici, Zeger, Schwartz, Burnett, Krewski, Bates (2007) Bayesian Model Averaging in Time Series Studies of Air Pollution and Mortality. J. of Toxicology & Environmental Health, Part A, 70: 1-5 Rebut K&T by showing that the data provide very little information on model choice and that you need to be careful in selecting candidate models and, if Bayes, priors Moral: Statistical wrappers can be used to package mischief ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 16

17 Improve Quantification of Uncertainty! NAS Panels and others repeatedly call for increased and improved quantification of uncertainty And for incorporating uncertainty into the policy-making process We can quantify the effect of and uncertainty associated with any working scenario (a model with some free parameters) We can do a variety of sensitivity analyses If we are lucky, an aggressive sensitivity analysis will show that the decision is robust, but that is usually not the case ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 17

18 There are limits to what we can and should do Model choice will always have unquantifiable aspects, absence of credible priors and little information on model choice Communicating too much uncertainty may work against achieving goals We recommend screening every 2 years, but it might be better to do it every 1 year, and yet some evidence suggests that every 3 years is best [:-( Lexington Mass school enrollment projections ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 18

19 How to deal with unquantifiable uncertainties? How to proceed with little information on model choice? We can bring in expert opinion/ knowledge on mechanism, genomics, discount rates,... But, how do we rate the credibility of an expert? We can conduct sensitivity analyses We can try for conservative approaches There are limits to all formalisms ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 19

20 It s never purely technical Statistical issues permeate much in the policy domain However, nothing is purely statistical, but essentially everything has statistical content There are real limits to stakeholder understanding And, frequently political or legal considerations will trump statistical performance A few examples... ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 20

21 Statistical Issues in Allocating Funds by Formula, 2003 The federal government allocates billions of dollars using formulas Medicaid, Highway Planning & Construction, Temporary Assistance for Needy Families, Title I school programs,... Some formula provisions appear illogical, for example small state minima Explained by, In the Senate each state has two votes Some, for example Title I, have thresholds and invite threshold-optimal estimates We developed such estimates, but these and other recommendations were (are?) moot due to a hold harmless provision Moral: Statistical optimality trumped by politics & predictibility ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 21

22 Small-area Income and Poverty Estimates (1997, 1998, 1999, 2000) Each fall, the Census Bureau produces these estimates and they are used for Title I and other allocations In the summer of 1997 an NAS panel had either to bless the current approach or advocate a new one We didn t want to bless, but we weren t ready fully to endorse our proposed approach ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 22

23 Small-area Income and Poverty Estimates (1997, 1998, 1999, 2000) Each fall, the Census Bureau produces these estimates and they are used for Title I and other allocations In the summer of 1997 an NAS panel had either to bless the current approach or advocate a new one We didn t want to bless, but we weren t ready fully to endorse our proposed approach So, the 50% solution: We recommended a 50/50, convex combination of the current and our proposed approach with the rationale that there needed to be a transition period Also, of course, we needed more time ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 23

24 Small-area Income and Poverty Estimates (1997, 1998, 1999, 2000) Each fall, the Census Bureau produces these estimates and they are used for Title I and other allocations In the summer of 1997 an NAS panel had either to bless the current approach or advocate a new one We didn t want to bless, but we weren t ready fully to endorse our proposed approach So, the 50% solution: We recommended a 50/50, convex combination of the current and our proposed approach with the rationale that there needed to be a transition period Also, of course, we needed more time External reviewers were fine with a convex combination, but asked that we justify the 50/50 weights ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 24

25 The COPSS/CMS White Paper Since 2008, the CMS has been publicly reporting condition-specific outcome measures such as risk adjusted all-cause mortality and readmission measures for AMI, HF and Pneumonia CMS uses hierarchical modeling to produce casemix risk-adjusted hospital-specific measures, that compare a hospital s performance to a national standard Standardized Mortality Ratio = 1.0 means the hospital performs at the national standard for its case-mix CMS has received considerable push-back from stakeholders as well as research communities ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 25

26 Principal Pushbacks The CMS approach 1 Fails to reveal provider performance variation: The shrinkage effect of the hierarchical modeling reduces the variation of the hospital performance and renders the information not useful for the consumers 2 Masks the performance of small hospitals: It is pointless to include the small hospitals in the calculation based on hierarchical modeling because the small hospitals would get a rate close to the national mean. The methodology neutralizes small hospital performance 3 Is difficult to communicate and explain: Stakeholders are familiar with the Observed/Expected output of standard logistic regression modeling. But, the approach adopted by CMS replaces the O with a shrinkage estimate (referred to as predicted in CMS documents) 4 There is a considerable lack of consensus regarding: Whether to use hierarchical models Whether to use a random-effects approach Whether to stabilize by shrinkage Whether to use hospital attributes to adjust shrinkage targets... ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 26

27 The COPSS-CMS Committee Consequently, CMS supported a committee appointed by the COPSS to address statistical issues identified by the CMS and stakeholders about CMS s approach to modeling hospital quality based on outcomes The charge:... provide guidance on statistical approaches... when estimating performance metrics, and consider and discuss concerns commonly raised by stakeholders [...] about the use of hierarchical generalized linear models in profiling hospital quality Coordinator & Committee Members Xihong Lin: COPSS Coordinator Arlene Ash, PhD: Representing the ASA Steve Fienberg, PhD: Representing the IMS Tom Louis, PhD: Committee Chair Sharon-Lise T. Normand, PhD: (Representing IBS-ENAR Thérèse A. Stukel, PhD: Representing the SSC) Jessica Utts, PhD: Representing IBS-WNAR ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 27

28 Process & Status Process Status Approximated the NAS process including COI disclosure, and external review of a draft report But, the report was to the CMS and not of the COPSS and so CMS has control over release of the report However, we do have pre-approval to publish an article and will have a session at ENAR in April Turned in our white paper on November 29, 2011 Have had a few conversations with CMS and the National Quality Forum Without our knowledge our white paper was posted on the NQF website CMS has not yet released/posted it, but will do so soon Morals: A long list! ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 28

29 Educating Policy Savvy Statisticians When is the right time to train and educate for policy involvement? The King Lear analogy What should be the curriculum? Need a solid understanding of Math. Stat. & Applied Stat. It is an absolutely necessary platform from which to leap into the policy miasma It is by no means sufficient: One needs to understand the science/policy context through case studies and experience We need to use the apprentice/intern model: Attendance at NAS meetings, NIH meetings, DSMB closed sessions, etc. ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 29

30 Educating Statistically Savvy Policy Makers, Implementers, Reporters, and Consumers Grab/create educational opportunities Read and distribute, Cohn, Cope, Cohn Runkle, News and Numbers, A Writer s Guide to Statistics, Third Edition (Wiley-Blackwell, 2011) Advocate and propagate statistical concepts and methods to all (future) stakeholders of all ages and roles Promote and support our discipline and colleagues, while maintaining high standards ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 30

31 What do we mean by success? That studies are designed, conducted, analyzed and reported using sound statistical principles That the evidence is used appropriately in making decisions This does not mean that priors and losses are out of the picture, but that decisions should be guided by the evidence That quantifiable and unquantifiable uncertainties are identified and communicated That communication to relevant decision-makers and stakeholders is honest and understandable Success can breed displeasure: we will never be pleased with all uses of statistics ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 31

32 Some Successes NRC: CNSTAT, CATS and much more The Prevention and Treatment of Missing Data in Clinical Trials UN office of statistics and similar offices in member countries Federal statistical agencies (e.g., Census, BLS,...), but... OMB Chief Statistician NIH, EPA, CDC FDA/CDRH & Bayesian designs and analyses Study sections, DSMBs, other advisory groups CASAC Policy organizations: Rand, Urban Institute, Mathematica,... Halothane, Value added modeling, Challenger, post hoc (should have been pre-hoc) Air Pollution Criterion documents and policy decisions Baggerly/Coombes and reproducible research AHRQ processes,... Emerging ASA presence & impact Small steps in the AAAS Many, many individual statisticians, including speakers at this workshop ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 32

33 Some frustrations and threats Relatively poor visibility Not at the table or not at it early enough for many, highly statistical activities Getting there requires Proactive and energetic promotion of the importance of our field and our willingness to participate Continuing (continuous) education of collaborators, administrators, policymakers, stakeholders Being at the right place at the right time Semantic challenges including Mathematics Biometrics and Informatics Insufficient funding to educate the next generation ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 33

34 Summary Many successes and many frustrations Many job opportunities and opportunities for positive impact We need to be eternally vigilant and energetic so we can elevate our prevalence and roles in policy arenas We need to be and be viewed as being excellent and essential We need to promulgate the health rather than the illness role of statistical thinking and practice Lead by example, ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 34

35 Summary Many successes and many frustrations Many job opportunities and opportunities for positive impact We need to be eternally vigilant and energetic so we can elevate our prevalence and roles in policy arenas We need to be and be viewed as being excellent and essential We need to promulgate the health rather than the illness role of statistical thinking and practice Lead by example, The respect conferred by others to statistics and statisticians will be no greater than the respect we convey and confer ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 35

36 THANKS ASA Statistics & Public Policy Conference T. A. Louis: Biostatistics in Policy page 36