Dd Illegal Overseas Absentee Ballots Decde the 2000 U.S. Presdental Electon? 1 Kosuke Ima 2 Gary Kng 3 March 23, 2004 1 We are deeply grateful to the many prvate ctzens of every poltcal strpe who took the tme to send us comments on ths paper. Thanks also to Jm Alt, Barry Burden, James Honeker, Doug Rvers, and Jonathan Wand for helpful dscussons; Assocate Edtor Henry Brady for many suggestons; and the Natonal Scence Foundaton (IIS-9874747), the Natonal Insttutes of Agng (P01 AG17625-01), and the World Health Organzaton for research support. Software to mplement the methods n ths paper s avalable at http://gkng.harvard.edu. 2 Assstant Professor, Department of Poltcs, Prnceton Unversty. (Corwn Hall, Department of Poltcs, Prnceton Unversty, Prnceton NJ 08544; http://www.prnceton.edu/ kma, kma@prnceton.edu, (609) 258-6601). 3 Davd Florence Professor of Government, Harvard Unversty (Center for Basc Research n the Socal Scences, 34 Krkland Street, Harvard Unversty, Cambrdge MA 02138; http://gkng.harvard.edu, Kng@Harvard.Edu, (617) 495-2027).
Abstract Although not wdely known untl much later, Al Gore receved 202 more votes than George W. Bush on electon day n Florda. George W. Bush s presdent because he overcame hs electon day defct wth overseas absentee ballots that arrved and were counted after electon day. In the fnal offcal tally, Bush receved 537 more votes than Gore. These numbers are taken from the offcal results released by the Florda Secretary of State s offce and so do not reflect overvotes, undervotes, unsuccessful ltgaton, butterfly ballot problems, recounts that mght have been allowed but were not, or any other hypothetcal dvergence between voter preferences and counted votes. After the electon, The New York Tmes conducted a sx month nvestgaton and found that 680 of the overseas absentee ballots were llegally counted, and almost no one has publcly dsagreed wth ther assessment. In ths paper, we descrbe the statstcal procedures we developed and mplemented for the Tmes to ascertan whether dsqualfyng these 680 ballots would have changed the outcome of the electon. We present a varety of new emprcal results that delneate the precse condtons under whch Al Gore would have been elected presdent, and offer new evdence of the strkng effectveness of the Republcan effort to prevent local electon offcals from applyng electon law equally to all Florda ctzens.
Gore Bush margn Ballots cast/receved by Nov. 7 2,911,417 2,911,215 Gore leads by 202 Late overseas absentee ballots a 836 1,575 Bush leads by 739 Total 2,912,253 2,912,790 Bush leads by 537 Table 1: Offcal results of the 2000 presdental electon n Florda. Source: Florda Secretary of State s offce. a Ballots counted from November 7th to 26th. 1 Introducton Many aspects of the 2000 electon were the subject of consderable meda attenton, ltgaton, and academc analyss n the uncertan month that followed the votng. In contrast, Florda s overseas absentee ballots were spared most of ths attenton and all ltgaton. Yet, they determned the electon outcome: If only the votes cast on electon day were counted, Al Gore would have beat George W. Bush by 202 votes and become presdent. Accordng to offcal results from the State of Florda, t took the overseas absentee ballots for Bush to outdstance Gore, whch he dd n the end by 537 votes (see Table 1). The extent to whch the law was followed n ths small but consequental part of the story escaped scrutny for some tme. After the electon was certfed, however, The New York Tmes conducted a sx month nvestgaton durng whch they retreved the envelopes n whch the ballots were maled, and searched for volatons of the law (Barstow and Van Natta, Jr., 2001). In one of the longest set of artcles ever publshed by The Tmes, they concluded that 680 of the overseas absentee ballots that had been counted by Florda countes unambguously volated one or more aspects of Florda electon law and, by any reasonable nterpretaton of the law, should have been dscarded. Indeed, after The Tmes story appeared, commentators and partsans from both sdes accepted these factual clams. 1 In comparson wth other features of the electon that have been studed, ths problem was not caused by old machnes or the nattenton of local electon offcals. It also does not rely on somehow nferrng the ntent of the voters. Rather, accordng to The Tmes, the overseas ballot problem was due to blatantly llegal actons on the part of local electon offcals, encouraged by Republcans, that had not been prevously notced. The Tmes argued that local offcals were nfluenced by the delberate poltcal strateges employed by the Bush campagn, and comparatve neglect by the Democrats. 2 They concluded that Under ntense pressure from the Republcans, Florda offcals accepted hundreds of overseas absentee ballots that faled to comply wth the state laws (Barstow and Van Natta, Jr., 2001). Were these 680 napproprately counted ballots enough to have thrown the electon to the wrong canddate? The Tmes hred us to fnd out. Our conclusons were presented as part of the story (Barstow and Van 1 The only excepton seems to have been Zelnck (2001). 2 The Democrats had planned to contest the absentee ballots, but Democratc Vce Presdental Canddate Joe Leberman on Meet the Press ended ths strategy when he explaned that he would gve the beneft of the doubt to ballots comng n from mltary personnel generally.... Al Gore and I don t want to ever be part of anythng that would put an extra burden on the mltary personnel abroad who want to vote. Gore backed hm up and left the Republcan strategy unchallenged. See Berke (2001). 1
Natta, Jr., 2001) and our methods were brefly descrbed n a sdebar (Barbanel, 2001). In ths paper, we dscuss n detal the methods we developed for ths project so that others mght use them for smlar problems. The stuaton calls for ecologcal nference: We observe the number of bad ballots n each of Florda s countes and the number of ballots cast and counted for each of the canddates. From these varables, and a varety of other auxlary nformaton, we try to nfer the total number of bad ballots that had been cast for each canddate and see whether ths s enough to make up for Bush s offcal 537 vote margn. Snce the partsan atmosphere surroundng publc dscourse on ths ssue was so hghly charged, we knew that our work would be subject to more than the usual academc scrutny, and so we sought out methods that were less vulnerable to partsan crtcsm. We therefore used three separate approaches nference wth no statstcal assumptons, wth a successon of sngle models that any partsan mght have consdered, and fnally wth Bayesan model averagng that enable us to average over all of these sngle models wth weghts beng ther relatve probablty of beng correct, as ndcated by the data. So that others can use the methods we ntroduce here to analyze other problems, we have ncluded all methods ntroduced here n the program EI: A Program for Ecologcal Inference. We estmate the probablty that Gore would have won the electon f the law had been followed n ths nstance. Ths probablty s small, but we show that wth mathematcal certanty t s greater than zero. Secondly, although our results suggest that t s unlkely that llegal overseas absentee ballots alone changed the electon outcome, we show that Bush s margn of vctory would lkely have been much narrower f those flawed ballots had not been counted. Ths supports the argument made by The Tmes that the flawed ballots favored Bush much more than Gore. We also present a varety of results that dd not appear n The Tmes artcle, ncludng the probablty that Gore would have won under varous hypothetcal scenaros, such as f Katherne Harrs had accepted Palm Beach county s recount, whch was submtted two hours late. In some plausble scenaros, the probablty that Gore would have won s nearly 100%. Fnally, and perhaps most nterestng, we present evdence that the propensty of local electon offcals to volate the law and accept bad ballots was substantally greater n countes where Bush strategsts beleved there were more absentee ballot support for Bush and tred to convnce electon offcals to accept bad ballots. Ths s consstent wth The Tmes thess and evdence that these local electon offcals bent to the wll of Republcan lobbysts. 2 Invald Overseas Absentee Ballots n Florda On July 15, 2001, The New York Tmes publshed an artcle, How Bush Took Florda: Mnng the Overseas Absentee Vote, as the result of ts sx-month nvestgaton on the 2000 electon. The Tmes reporters descrbe the detals of the Bush campagn effort to secure vctory by pressurng selected local electon offcals to count nvald overseas absentee ballots n Florda. In partcular, Republcans focused on mltary ballots and the countes where Bush had hs strongest votng base. For example, n countes such as Escamba and Santa Rosa, Bush lawyers argued that every vote cast by Amercans n unform should be counted, regardless 2
of the letter of the law. In Democratc countes, Bush s lawyers argued exactly the opposte that local electon offcals must follow the letter of the law and dsqualfy any ballot not meetng the rules. Accordng to The Tmes, ths unequal pressure led to unequal treatment by local offcals of overseas voters. That partsans would pursue ther nterests creatvely, relentlessly, and even nconsstently n dfferent places s nether a novel clam nor remotely llegal. That local electon offcals would respond to ths pressure by treatng voters unequally s a more serous clam. The Tmes vew The result was unequal treatment of ballots wth the same flaws contradcts statements by Florda Secretary of State, Katherne Harrs, that the rules were appled unformly. It also would seem to volate the Equal Protecton Clause of the U.S. Consttuton, whch was part of the stated grounds under whch the Unted States Supreme Court n Bush v. Gore stopped the manual recounts. The 680 ballots that The Tmes judged as flawed fell nto one or more of these categores (Barstow and Van Natta, Jr., 2001): 3 344 ballots had late, llegble or mssng postmarks (postmarks must ndcate that the ballot was cast on or before electon day); 4 183 ballots wth Unted States postmarks (overseas absentee ballots must bear foregn postmarks); 169 ballots were receved from voters who were not regstered, who had faled to sgn the envelope, or who had not requested a ballot; 96 ballots lacked the requred sgnature or address of a wtness; 19 voters cast two ballots, both of whch counted; 5 ballots were receved after the Nov. 17 deadlne but counted anyway. If we knew for whch canddate the llegal ballots were cast, we would mmedately know ther effect on the electon. However, the secret ballot makes ths mpossble n most cases. The secret ballot was mplemented n ths case by separatng the envelope, wth all the nformaton above, from the ballot contaned nsde the envelope once the latter was counted. Thus, we only have access to these envelopes, the county n whch they were counted, and county-level data on the number of bad ballots and the number of ballots cast for Gore and Bush. Table 2 llustrates the estmaton problem at the state level. The queston mark ndcates the unknown quanttes to be estmated. The table llustrates that whle we know the aggregate number of nvald and vald ballots as well as the total number of votes each canddate obtaned from overseas absentee voters, we do not know ther composton, whch s the goal of the analyss. Analogous contngency tables also exst for each of the 67 Florda countes, and the same ecologcal nference problem exsts n each. We also receved three other knds of data for each county. Frst, from voter regstraton records, we have data about each overseas absentee voter, ncludng ther sex, race, party regstraton, and whether they were mltary personnel or cvlan. Second, for comparatve purposes we also have data avalable for electon-day voters n the 67 countes. Fnally, The Tmes also provded us ndcator varables for four regons n Florda and 3 Photographs of the bad ballots make the determnaton of flaws unambguous; for examples, see http://www.nytmes. com/mages/2001/07/15/poltcs/absentee/nat_absentee_count_ndex.html. 4 At tmes, the Republcans argued that mltary ballots dd not need a postmark because the law allowed those n the mltary to send mal wthout postage. However, accordng to The Tmes, Florda state law clearly requred overseas ballots to be postmarked or sgned and dated by electon day (Barstow and Van Natta, Jr., 2001). 3
Gore Bush Others total nvald ballots??? 680 vald ballots??? 1810 836 1575 79 2490 Table 2: The Ecologcal Inference Problem n Florda.? ndcates the unknown quanttes to be estmated. nvald ballots vald ballots Gore β bad β good T Bush 1 β bad 1 β good 1 T X 1 X Table 3: Ecologcal nference for nvald overseas absentee ballots n Florda some other measures. We use ths extra nformaton to mprove our ecologcal nferences. 3 Ecologcal Inference for Flawed Ballots Table 3 presents our notaton. For each county ( = 1,..., 67), we denote the proporton of nvald ballots among all overseas absentee ballots as X, and the total number of overseas absentee ballots whch were counted as N. We let Gore s proporton of the vote be T. To smplfy presentaton, we combne the votes for Bush and the other mnor canddates as Bush votes. 5 Whle each of these quanttes are observed, we denote unobserved quanttes wth Greek letters: β bad vald ballots cast for Gore, respectvely. Although β bad and β good and β good represent the proportons of nvald and are used for the estmaton, our ultmate quantty of nterest s Bush s margn after droppng the nvald absentee ballots. To defne ths quantty, frst defne the statewde fracton of bad ballots that went to Gore β bad as the weghted average of the ndvdual county quanttes. 6 Thus, Bush s margn = offcal margn [Bush s bad ballots Gore s bad ballots] = 537 [(1 β bad )680 β bad 680] = 1360β bad 143. (1) Once we estmate ths quantty, we can also estmate the probablty of Gore s vctory, Pr(Bush s margn < 0), whch s the man quantty of nterest. 7 3.1 Analyss Wthout Statstcal Assumptons The the parameters n Table 3 follow an accountng dentty T = β bad X + β good (1 X ) (2) 5 Although our presentaton always nvolves only the Bush/Gore choce, our emprcal results usng determnstc bounds n Secton 3.1 ncludes the possblty of bad ballots havng been cast for mnor party canddates. We handle mnor partes n our statstcal analyses by gnorng the problem at frst and then conductng senstvty analyses n Appendx B; snce votes for mnor party canddates only total 3 percent, we fnd, as expected, that they have a very small effect on the overall result. Other analyses (not shown) usng more computatonally ntensve technques desgned to model these choces separately confrm these results (see Rosen et al., 2001). 6 The weghted average s β bad = P 67 =1 N β bad / P 67 =1 N. 7 Note that β good s not used n Equaton 1 but s necessary as an ancllary parameter durng estmaton. 4
Total Gore s votes Bush s votes Others nvald offcal vald ballots offcal vald ballots offcal vald ballots County ballots counts mn max counts mn max counts mn max Escamba 102 47 0 47 154 52 106 7 0 7 Santa Rosa 55 16 0 16 65 10 28 2 0 2 Baker 1 0 0 0 1 1 1 0 0 0 all countes 680 836 309 831 1575 907 1447 79 0 79 Table 4: Analyss of bounds for the state and selected countes. whch s generated by the aggregaton process, and therefore always holds exactly wth no stochastc term. It also mples a determnstc lnear relatonshp between the two unknown parameters (Duncan and Davs, 1953), β good = T 1 X X 1 X β bad, (3) whch traces out what Kng (1997) calls a tomography lne. In addton, before we observe X and T n any county, we also know that β bad and β good are each between 0 and 1. Once we observe X and T, we can narrow the bounds further (by projectng the lne n Equaton 3 to the two axes). Thus, wthout any statstcal assumptons, we can derve the upper and lower bounds of β good and β bad for each county, whch n turn mply the bounds for our quantty of nterest, Bush s margn after droppng flawed overseas absentee ballots. Table 4 shows how the analyss of bounds can be very powerful n some stuatons. For example, Escamba s one of the countes where many nvald ballots were found. At the same tme, ths county s one of Bush s strongholds: about 76 percent of overseas absentee ballots for ths county were counted for Bush. In Escamba, there were 208 total votes, 47 for Gore, 154 for Bush, and 7 for others. Of these votes, 102 ballots were nvald. Because Bush had 154 total votes, t s possble that all 102 of these nvald ballots were Bush s votes. Ths produces the lower bound on the number of vald Bush s votes, namely 52. The upper bound occurs when Gore and other canddates get assgned the maxmum number of nvald ballots. Snce Gore had 47 votes and other canddates got 7 votes, the maxmum number of nvald ballots that can be assgned to them s 54, leavng 48 nvald votes for Bush. Hence, the maxmum number of vald votes for Bush s hs offcal count of 154 mnus these 48, whch s 106. Fnally, Baker county llustrates why sometmes the secret ballot s not really secret. In ths county, only one absentee ballot was cast and also was found to be nvald. Hence, we know from the total tally of absentee ballots n ths county one vote for Bush that ths person voted for Bush and that t was an nvald ballot but ncluded n the offcal count. 8 From these county level bounds, we derve aggregate bounds for the total number of nvald ballots for each canddate at the state level. The result shows that at least 8 percent, or 128 votes of Bush s 1,575 absentee ballots, should not have been counted, whereas for Gore the mnmum number of nvald ballots s only 5 out of hs total 836 votes (0.6 percent). Furthermore, Bush could have napproprately benefted from 8 Indeed, the name, address, and ndvdual vote cast of all people n countes, lke Baker, that cast all ther absentee ballots for one canddate, are on the publc record. Ths s because the bounds have zero wdth whenever ether X or T s zero or one. 5
up to 668 out of the 680 nvald ballots. The most sgnfcant concluson from ths analyss s that we cannot exclude the possblty that Gore actually won the electon. That s, wthout makng any assumptons other than that The Tmes codng decsons were correct (and agan, we saw no objecton to them n the meda dscusson that followed ther story), the 537 Bush margn now changes to somewhere from a 126 vote vctory for Gore to a 936 vote vctory for Bush. Once the ballots were removed from the envelopes, Amerca forever gave up the possblty of knowng for certan who won the most votes n the 2000 electon. 3.2 Statstcal Analyss wth a Sngle Model Specfcaton The lack of any statstcal assumptons puts the analyss n Secton 3.1 on extremely frm footng. In fact, measurement error asde, the conclusons there contan no nferental uncertantes at all, whch s of course qute unusual for socal scence research. If the resultng bounds excluded the possblty of one canddate wnnng, our analyss would end rght here. Unfortunately, all we know from the bounds s that there exsts a possblty that ether Gore or Bush receved more votes. Indeed, even f all but a tny pece of the bounded nterval reflected a Bush (or a Gore) vctory, t would provde no nformaton about the probablty that a partcular canddate won other than that zero s excluded (snce the bounds alone do not mply any probablty dstrbuton over the nterval). Snce ths probablty was the quantty of nterested for our project, the Tmes needed us to go further than the bounds and to make an nference about the probablty that about 90% of the bad ballots went to Bush, whch f corrected would have produced a Gore vctory. For that quantty, the bounds alone are n suffcent. In order to learn more about who actually won the electon the lkely margns of vctory wthn the determnstc bounds the only opton s to add some statstcal assumptons. Wth these assumptons, we can make probablstc statements about our quanttes of nterest. The problem wth any model-based statstcal analyss, of course, s that there mght be a dsagreement about the assumptons to be made, and so the prce of the more precse conclusons that follow s the addtonal uncertanty due to model specfcaton. Ths s a common problem n socal scence research, but t s partcularly salent when attemptng to provde ndependent nonpartsan advce n the mdst of one of the most hghly charged partsan debates n modern tmes. Our approach to ths problem s to formally ncorporate uncertanty due to model specfcaton nto our fnal estmate. We use the class of ecologcal nference models gven n Kng (1997), whch has come to called EI after the software that mplements them. To begn, we summarze graphcally the bounds obtaned n Secton 3.1. For each Florda county, we show all possble values of β good and of β bad. As t turns out, the data tell us more than merely the bounds on each parameter separately snce Equaton 3 says that f β good of ts bounds, β bad s near the top must be near the bottom of ts bounds. In fact, the two are perfectly correlated and fall on a lne defned by that equaton. To llustrate, we plot one lne for each county n Florda n the tomography plot of Fgure 1, whch merely re-expresses the data n the form closest to the answers we seek. 6
β good 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 β bad Fgure 1: Tomography plot for nvald overseas absentee ballots. β bad and β good are the proporton of Gore s nvald and vald ballots, respectvely. Each lne traces out the possble values of the β bad, β good pont for each county. The sold lne s for Escamba county. All of the above s as far as t s possble to go wthout makng statstcal assumptons. Next, we add three assumptons, all condtonal on X and a specfed set of control varables Z. We could begn wth the assumpton that β bad and β good are the same for all countes, whch s essentally Goodman s regresson. Ths would be enough to dentfy the model, but t would be a very strong assumpton, and ndeed n our applcaton t can be rejected wth certanty by merely examnng Fgure 1. Thus, nstead of assumng that β bad and β good are the same over countes, we assume that they come from the same dstrbuton (the truncated bvarate normal dstrbuton, wth truncaton kept to the square n the fgure). The dea s that whatever the values of the unknown parameters from Florda s 67 countes on ther respectve tomography lnes, they all have somethng n common snce they are all n the same state, subject to almost the same electoral campagn, etc. Dfferences of any sze across the countes n these parameters are thus allowed so long as they are ether random and ft the dstrbutonal assumpton or they are systematc and taken nto account by control varables. The man constrants added by ths assumpton s that the bvarate densty s unmodal (whch s the formalzaton of the assumpton that the countes have somethng n common ) and that all volume under the densty appears over the square represented by the tomography plot. Volatons of ths dstrbutonal assumpton do not seem to affect the quanttes we need for ths applcaton (see Kng, 1997: Table 9.2). The second assumpton s that the absence of resdual spatal correlaton n T after takng nto account X and Z. Kng s ecologcal nference model has been shown to be relatvely nsenstve to anythng but extreme levels of spatal autocorrelaton (Kng, 1997: Table 9.1, Kng, 2000), but we make ths assumpton even more plausble by ncludng tests wth covarates that tap nto Florda s regons and other spatal features. The fnal assumpton states that the two unknown quanttes, β bad and β good, are ndependent of X, gven Z. For example, f more bad ballots cast for Gore, β bad, came from countes wth more bad ballots overall, X, then ths assumpton would be volated unless Z ncluded varables that suffcently controlled 7
for ths relatonshp. Ths s the most crtcal of the three assumptons, and so the valdty of the analyss depends crucally on the content of Z. Each of these three assumptons can be modfed or relaxed by the ncluson of dfferent covarates, Z, and so the man model uncertanty that s assumed presently s Z. 3.3 Statstcal Analyses that Acknowledge Model Uncertanty Senstvty to model specfcaton n quanttatve poltcal scence s perhaps most commonly seen when mnor changes among the explanatory varables n regresson-type analyses result n large dfferences n the estmates. The endemc nature of such model dependent nferences makes a decson to base nferences on a sngle model hghly dubous n many stuatons. Ths s all the more so n ecologcal nference where model dependence s frequently an ssue. Yet, almost all exstng applcatons of ecologcal nference use a sngle specfcaton. Indeed, few use any explanatory varables, Z, at all. The partsan nature of the controversy n whch we were provdng advce makes the ssue of model dependence especally salent, although t s not markedly dfferent from other applcatons. We began by followng the most common procedure of estmatng many models (.e., wth dfferent Z) and assessng the degree to whch our ultmate quanttes of nterest depend on the specfcaton. Ths was nformatve but nsuffcent, snce our task was to provde a sngle nference wth a pont estmate and confdence nterval. Three basc ways of drawng a sngle nference n the presence of model uncertanty exst. Some researchers persst n choosng a sngle model, perhaps on the bass of qualtatve arguments about ts merts, and draw nferences assumng ts veracty. Ths optmstc approach typcally overestmates the degree to whch the researcher s certan of the correct model specfcaton, and hence typcally gves based estmates and overly narrow confdence ntervals. Other researchers use a formal model selecton crteron, such as stepwse regresson, Mallow s Cp, and AIC, to pck the best model. Although these procedures are napproprate when estmatng causal effects, they are reasonable when the quantty of nterest s predctve, such as ecologcal nference. Unfortunately, even n these stuatons the result of applyng these crtera s one model, whch also gnores model uncertanty. An approach now wdely recognzed to be superor to standard model selecton crteron s Bayesan model averagng, whch we apply to our ecologcal nference problem. The basc dea s to estmate a large number of potental models and to take the weghted average of ther results, wth weghts based on probablty that a model s correct. The correctness of the model s not assumed ex ante; nor s t merely based on goodness of ft; t s nstead calculated from the data va Bayesan analyss. The key result of Bayesan model averagng s that the resultng nferences (1) are more accurate than those produced by standard model selecton crteron, (2) formally ncorporate model uncertanty, and (3) outperform any of the ndvdual models that are averaged over. The last property does not depend on whether the true model generatng the data les nsde or outsde models averaged over. That s, nferences from Bayesan model averagng always outperforms any ndvdual model consdered (e.g., Madgan and Raftery, 1994). 9 Model 9 Ths result s smlar to nsghts from the closely related lterature on commttee methods (Bshop, 1995), although surprsngly 8
averagng thus allows one to consder a wde range of models, whle stll producng one set of results. Of course, one can never cover the entre model space, whch s normally of nfnte sze, snce Bayesan model averagng only allows one to nclude a fnte number of models. Therefore, the resultng nferences can always be mproved by addng addtonal models, no matter how many models have already been ncluded. Ths s not a unque feature of Bayesan model averagng snce t s always possble to come up wth a better model for any statstcal analyss where the correct model s unknown to researchers. However, Bayesan model averagng offers a sgnfcant mprovement over the usual approach of basng nferences on one assumed model, no matter how that model s chosen. Bayesan model averagng s especally mportant n our applcaton snce poltcal scentsts have rarely studed absentee ballots and we therefore have lttle pror theory wth whch to assst n model specfcaton. The procedure thus enables us to conduct an analyss wthout havng to defend one partcular specfcaton, or even a small set of specfcatons. We came up wth our lst of models by talkng n detal to reporters and partsans on both sdes. We then formalzed every ntuton any of them mentoned n an ecologcal nference model (by defnng Z) and ncluded every one n our analyss. We also added several other models we came up wth ndependently. Our search for models dentfed 31 possbltes for Z, ncludng ncludng race, sex, and party regstraton for the overseas absentee voters as well as models based on 24 county level electon and demographc varables. We also nclude a model wth no covarates and 3 models wth X as the covarate for the mean of β bad, β good, and for both. 10 Each of our models ncludes at most two covarates. In part ths s because no one ncludng journalsts and academcs proposed a model that clearly was defned by more. But more mportantly, we know that better predctons can be obtaned when not overfttng the data wth many covarates. Hoetng et al. (1999) and Madgan and Raftery (1994) and many others have shown that Bayesan model averagng almost never puts much weght on such models, and predctve nferences (unlke some the lteratures have relatvely few cross-ctatons. See also Rosen, Jang, and Tanner (2000) and Robert (1996). See Hoetng et al. (1999) for a general ntroducton to Bayesan model averagng. Raftery and Zheng (2003) derve the optmalty of ts longrun performance. See Bartels (1997), Bartels and Zaller (2001), and Erkson, Bafum and Wlson (2001) for poltcal scence applcatons that use an approxmaton to formal Bayesan model averagng. 10 A lst of all 31 models follows: (1) no covarate, (2) X for β bad, (3) X for β good, (4) X for both β bad and β good, (5) mltary absentee voters, (6) regstered Republcan absentee voters, (7) regstered Democratc absentee voters, (8) female absentee voters, (9) Whte absentee voters, (10) Black absentee voters, (11) Hspanc absentee voters, (12) rejected mltary absentee voters, (13) rejected regstered Republcan absentee voters, (14) rejected regstered Democratc absentee voters, (15) rejected female absentee voters, (16) rejected Whte absentee voters, (17) rejected Black absentee voters, (18) rejected Hspanc absentee voters, (19) Democratc vote share among resdents, (20) vote share of Republcan canddates, (21) vote share of other canddates, (22) regstered Democratc resdents, (23) regstered Republcan resdents, regstered Black Democratc resdents, (24) proporton of votng age populaton not regstered, (25) Black regstered Democrats, (26) Black regstered Republcan resdents, (27) acceptance rato of overall absentee ballots, (28) rato of nvald absentee ballots, (29) Panhandle Florda regonal ndcator varable, (30) Southern Florda regonal ndcator varable, (31) corrupton ndcator. All the covarates except ndcator varables are entered as a rato varyng from 0 to 1. Except the frst three models, the covarate was used to model the condtonal untruncated mean of both parameters, β bad and β good. Models (5) to (11) are based on nformaton about the absentee ballots and so dfferent varables were avalable regardng the nvald and vald ballots; we used the former group to predct β bad and the latter to predct β good. We reran each model wth dfferent startng values to verfy that we found the global maxmum. We also examned each of the tomography plots wth confdence regons to search for outlers or bad model fts. In addton, we plotted E(T X) or E(T X, Z) by X or Z, and checked whether the observed T fell wthn the (say) 90% confdence nterval 90% of the tme. 9
causal nferences) are typcally better wth parsmonous specfcatons. 11 Although we know of no applcaton of ecologcal nference that uses more than a sngle explanatory varable (and almost all applcatons use none), we also tred expandng our setup from 0, 1, and 2 varable models to also nclude several 3 and 4 varable specfcatons. Although some of these hghly computatonally ntensve models had large enough estmated weghts to be meanngful, ncludng these n the Bayesan model averagng procedure dd not apprecably change our substantve estmates. Of course, we have also omtted an nfnte number of other possble models from the set we average over, and t s possble that future researchers wll fnd and nclude a model we excluded that would change our emprcal conclusons. For example, we exclude all models wth more than four explanatory varables, as well as all nteractons and all models based on data we do not have. Our Bayesan model averagng results are known to be better than any ndvdual model among those we average over, but our results could be overturned f someone fnds a plausble model to add that turns out to have a hgh probablty of beng correct and leads to dfferent nferences. 4 Emprcal Results 4.1 The Probablty That Gore Would Have Won Wthout Bad Absentee Ballots Fgure 2 portrays the posteror dstrbuton from our analyss of Bush s margn of vctory f the bad ballots had not been counted (the hstogram of 1000 draws from the posteror). Note frst that, as requred by the procedure, all area for ths dstrbuton s contaned wthn the bounds we found for ths quantty of 126 to 936. The weght of the statstcal evdence wthn these bounds clearly demonstrates that Bush benefted consderably by the bad ballots, and removng them thus takes away from hs margn. Ths s evdent n the fgure because almost all of the area of the hstogram of posteror probablty falls to the left of the offcal margn of 537. The mean margn of vctory for Bush wthout the bad ballots s only 251 votes. The fgure also portrays the probablty that Gore actually won the electon by the area under the curve to the left of zero. Ths s only about 0.2 percent, ndcatng that Gore probably would not have won, even f the bad ballots had been dscarded. 4.2 Other Counterfactuals Whle our results ndcate that t s unlkely that nvald overseas absentee ballots alone would have changed the electon outcome, the llegally counted ballots could have had a much more a sgnfcant effect when combned wth slght changes n decsons regardng the manual recounts. We show ths result by frst focusng on several scenaros about the two key countes where a manual recount was conducted, Mam Dade and Palm Beach. In Mam Dade county, electon offcals decded to stop the manual recount when 11 Multple covarates uncorrelated wth X causes no dentfcaton problems n EI. Ecologcal nference models that do not ncorporate nformaton from the determnstc bounds (such as n Goodman, 1953) are not dentfed when ncludng X or varables related to X (Kng, 1997: 42). Thus, to the extent that models that ncorporate the bounds, such as EI, are estmable when ncludng X or covarates that are related to X, the nformaton that makes ths possble s comng from the bounds. Predctons about the quanttes of nferences n EI are not often greatly affected by ncludng more than one covarate at a tme. 10
Frequency 0 50 100 150 200 Gore wns Bush wns offcal margn 537 100 0 100 200 300 400 500 600 Bush s margn of vctory Fgure 2: Posteror dstrbuton of Bush s margn of vctory wthout the 680 nvald overseas absentee ballots they made the judgment that they could not meet the recount deadlne, set by the Florda Supreme Court, 5p.m. Sunday, November 27. The partal manual recount gave a net gan of 157 votes to Gore. In Palm Beach county, they also could not fnsh the manual recount, but they submtted the result of the partal recount just before the deadlne, whch would have gven Gore a net gan of 192 votes for Gore. Later that day, Palm Beach electoral offcals reported the result of the complete recount to Katherne Harrs. She rejected ths complete recount as well as the partal recount and dd not nclude them n the certfed offcal tally, thereby denyng Gore a total of 349 votes (Purdum, 2000). The panel of Table 5 marked actual recounts presents our predcton for Bush s margn and Gore s probablty of vctory n stuatons where the nvald overseas absentee ballots had been rejected and the recounts n one or both of these countes had been ncluded n the fnal tally. For example, f the recounted votes n Mam Dade and Palm Beach had all been counted, Gore would have won wth a 0.82 probablty, wth the uncertanty n ths number comng only from our analyss of the bad overseas absentee ballots. If only the Palm Beach votes had been counted, Gore would have won wth 0.29 probablty. To put t one way, the massve dfferences n the probabltes from 0.002 to 0.82 for a Gore vctory were all due to the decsons of Katherne Harrs. Of course, these decsons could have been overturned by the courts, and the canddates could have nfluenced them f they had requested statewde or dfferent types of recounts. In the last panel of Table 5, we consder counterfactuals where the nvald overseas absentee ballots had not been counted and electon day votng recounts had occurred n varous ways, as suggested by a study conducted by a consortum of meda organzatons (Fessenden and Broder, 2001). For example, ths analyss shows that f the U.S. Supreme Court had not stopped the recount n Bush v. Gore, the vctor would have changed wth only a 1% probablty. However, f Gore s formal request that Broward, Mam Dade, Palm Beach, and Voulusa countes be recounted had been granted, then he would have been elected wth a 73% 11
Bush s margn Prob(Gore Wns) Invald overseas ballots alone 251 0.002 Actual recounts Mam Dade partal recount 94 0.19 Palm Beach recount 59 0.29 Mam Dade and Palm Beach 98 0.82 Meda recounts No U.S. Supreme Court decson a 242 0.01 Vote adjustments b 227 0.21 Gore s request granted c 26 0.73 hangng chads and dmples d 358 > 0.99 only fully punched ballots e 366 > 0.99 each county s standard f 422 > 0.99 Table 5: Estmated margn and probablty of vctory f the nvald overseas absentee ballots had not been counted along wth selected other counterfactuals (each of whch also excludes the nvald absentee ballots). a What would have happened f the U.S. Supreme Court had not stopped the manual recount. b Vald votes found by county offcals after offcal vote certfcaton c Gore requested that Broward, Mam-Dade, Palm Beach, and Voulusa countes be recounted. d Recountng all countes usng the standard that any hangng chad or dmple was counted. e Only fully punched ballots were recounted n all countes. f A recount of the entre state, usng the standards adopted by each county. probablty. If the entre state had been recounted, accordng to almost any standard for judgng the punch cards, Gore would have won electon wth a very hgh probablty. 4.3 Indrect Evdence of Local Electon Offcals Respondng to Republcan Pressure Sx months of ntervews and archval research on the ground n Florda and elsewhere led reporters from The New York Tmes to conclude that, the Republcans mounted a legal and publc relatons campagn to persuade canvassng boards n Bush strongholds to wave the state s electon laws when countng overseas absentee ballots.... Ther goal was smple: to count the maxmum number of overseas ballots n countes won by Mr. Bush, partcularly those wth a hgh concentraton of mltary voters, whle seekng to dsqualfy overseas ballots n countes won by Vce Presdent Al Gore. The Tmes clamed that as a drect result of ths pressure, canvassng boards n about a dozen Republcan-leanng countes had reconvened for a second round of countng. In each place, longstandng electon rules were bent and even gnored. Boards counted ballots postmarked as many as seven days after the electon, ncludng some from wthn the Unted States. They counted two ballots sent by fax. Offcals n Santa Rosa county even counted fve ballots that arrved after the Nov. 17 deadlne. Agan and agan, electon offcals crossed out the words REJECTED AS ILLEGAL that had been stamped on ballot envelopes. If these clams are correct, we ought to be able to fnd evdence of them n our data. We conduct two tests. In the frst, we dvde Florda s countes nto three categores the sx countes mentoned explctly n The Tmes story where the Republcans pressured offcals to count llegal ballots, the four countes mentoned 12
mltary Republcan Bad ballot Bad Ballots ballots vote acceptance a counted for Bush b all ballots Republcan pressure to count Coller 46.7% 65.6% 53.7% 64.5% 60 Duval 83.8 57.5 62.3 67.8 637 Escamba 88.6 62.6 64.2 80.3 272 Okaloosa 88.9 73.7 42.0 69.4 189 Pasco 62.3 48.0 60.5 76.4 53 Santa Rosa 90.3 72.1 84.6 84.4 93 Average 83.4 60.0 61.5 74.3 1304 Countes not mentoned by The Tmes Average 67.6 51.8 30.0 71.5 1751 Republcan pressure not to count Alachua 46.8 39.8 12.5 54.5 77 Broward 46.9 30.9 21.8 54.3 213 Mam Dade 44.4 46.3 11.7 57.1 306 Palm Beach 45.3 35.3 40.7 56.2 53 Average 45.6 38.1 17.2 55.4 649 Table 6: Countes classfed by whether The New York Tmes reported evdence of Republcan pressure to count or not count the overseas absentee ballots, compared to an average for the remanng countes not mentoned. Averages are weghted by the number of ballots. a The percent of bad ballots that arrved wth local electon offcals and were ncluded n the offcal count. b Ths column s estmated by our Bayesan model averagng ecologcal nference procedure. where Republcans pressured local electon offcals not to count the ballots, and the remanng countes whch were not mentoned. We then compute varous statstcs for these three categores and present them for comparson n Table 6. (The results n ths table were not avalable to the reporters before ther artcle appeared and so Table 6 does represent an ndependent test.) The evdence strkngly supports The Tmes account of events. The frst two columns of Table 6 report on the characterstcs of the county, nformaton avalable to Republcan strategsts before they started lobbyng. Wth the excepton of two countes wth very few absentee ballots, the countes dentfed as areas where the Republcans focused ther efforts to count ballots were those wth large populatons of mltary personnel and Republcan voters. Smlarly, the countes The Tmes dentfed as places where Republcans dscouraged the ballots from beng counted had consstently fewer mltary personnel and Republcan voters. The result of the Republcan efforts also appears to have been successful. A larger fracton of bad ballots were counted n all countes where Republcans tred to get them counted than the average, and a smaller fracton than the average were counted n every county where the Republcans tred to have them not counted. The fracton of bad ballots accepted that had been cast for Bush also supports the same theory: Fewer of the counted bad ballots had been Bush voters when the Republcans tred not to have ballots counted than n every county where the Republcans tred to have them counted. The Tmes report also helps explan some nterestng varatons n ths table. Frst, we would have expected more Bush votes among the bad ballots than we found n Duval county because t had so many 13
Bush s margn frst Posteror model (95 % C.I.) dfference probablty Bayesan Model Averagng 251 (69, 468) Indvdual models Regstered Repub. absentee voters, 6 a 269 (97, 475) -52 0.565 Dem. vote share, 19 232 (69, 448) 3 0.239 Black absentee voters, 10 231 (69, 440) -2 0.102 Whte absentee voters, 9 123 ( 18, 315) -23 0.033 Regstered black Repubs., 25 229 (62, 441) -6 0.021 Accepted absentee ballots, 27 218 (62, 409) 4 0.004 Table 7: Estmates of Bush s margn of vctory after droppng the nvald overseas absentee ballots overall and for the sx component models wth the hghest posteror model probabltes among the 31 models estmated. The frst dfferences represent the ncrease or decrease n Bush s estmated margn when the value of the covarate ncreases by 10 percentage ponts. a Each model s dentfed n the table by the covarate ncluded, followed by the model number we assgn to each n Footnote 10. mltary personnel. However, The Tmes reported that an electon offcal on the Duval county canvassng board held the lne on countng ballots wth mssng postmarks. Smlarly, Pasco county has relatvely low numbers of mltary ballots and a small Republcan vote share. So we mght expect that ths county to have had relatvely few of the bad ballots beng cast for Bush. However, the story also descrbed the unusually strong Republcan pressure appled n ths county: It looks to me lke we ve got a lot of pressure here, Judge Robert P. Cole, charman of the Pasco board, sad as he faced a throng of cheerng Republcans and more than a dozen Bush representatves [and no offcals from the Gore campagn]. Our quanttatve results are certanly consstent wth ths qualtatve evdence. We also look for ndrect evdence of local electon offcals succumbng to pressure from Republcan Party offcals by examnng the posteror probabltes of each of the 31 component models we ncluded. Generally, f The Tmes hypothess s rght, we would expect that the covarates that have the bggest effects would be related to where Republcans tred hardest to nfluence local offcals. If they were as ratonal and delberate as The Tmes suggested, these would be countes where they expected the largest numbers of bad ballots that, f counted, would help Bush s cause. Obvously, we have no such varable, but we do have a varety of varables related to ths. Table 7 gves the top sx models lsted n order of the posteror probablty of beng correct. In the top sx, two have the largest effects and both are consstent wth the theory: The more absentee voters regstered as Republcans, and the more whte absentee voters n a dstrct, the more bad ballots were cast for Bush (the negatve sgn ndcatng that Bush s lead s reduced when these ballots are not counted). The other covarates have comparatvely small effects. The large varaton n our predcton for Bush s margn across the sx models n Table 7 emphaszes a clear advantage of our Bayesan model averagng procedure. The varaton results from the large degree of model dependence n these data (because the data have farly wde bounds). For example, the specfcaton wth whte absentee voters gves a confdence nterval whch, when consdered n solaton from the other 14
models, would not enable us to reject the hypothess that Gore won f only the overseas absentee ballots had been rejected. Ths s obvously qute dfferent from our overall result of only a 0.2 percent probablty that Gore won. Snce dfferent specfcatons yeld very dfferent nferences, an analyst havng to choose one model would be n the untenable poston of havng to defend choces wthout a lot of pror evdence. Bayesan model averagng offers a way around ths common problem. Instead of results jumpng dramatcally from one specfcaton to the next, nferences resultng from Bayesan model averagng do not change as much when new models are added to the specfcaton, unless they have especally large probabltes of beng true. Of course, we cannot get somethng for nothng. Our procedure s an mprovement over straght EI because we only need to assume that one of our 31 models, or some combnaton of our 31 models, contans somethng close to the rght model. Ths s n contrast to the usual approach where we merely get one model from whch to choose, but t s not a panacea: although f someone comes up wth a new dea for a model, we can nclude t, but f none or no combnaton of those we consult comes close to the rght model, then our procedure wll obvously fal to gve vald answers. 5 Concludng Remarks Counterfactual analyss s normally dffcult, and especally so when the subject of the nference s far from the factual evdence. When the counterfactual s very close to the data, however, we stand an especally good chance of makng vald nferences (Kng and Zeng, 2001; Lebow, 2000). The counterfactuals n the case of Florda are especally clear and could have happened easly, whch makes the results of ths case study somewhat more certan than usual. If the problem of the overseas absentee ballots had been ltgated and the law appled equally n every county (as Bush v. Gore requred of the votes cast on electon day), the bad ballots mght very well have been dsqualfed. In ths stuaton, although Gore probably would have lost, we conclude that no one wll ever be able to say wth certanty who would have won the Amercan presdental electon f all Amercan laws had been followed. Also, f the Florda Secretary of State had dfferent vews on ssues that were at least somewhat open to dscreton, the outcome of the electon mght very well have changed. Of course, a few dfferent decsons by the canddates on vsts to Florda, campagn spendng, Elan Gonzalos, or any of a varety of other ssues mght also have produced a dfferent outcome. Our results also provde ndrect, but strong and ndependent support for the thess that local electon offcals bent to the persuasve efforts of Republcan strategsts to follow the law n Gore countes and break t n Bush areas. Fnally, we thnk ths paper also provdes an especally good example of the use of Bayesan model averagng. We have developed the applcaton of t to the ecologcal nference model and offer computer code for others to use t. Bayesan model averagng s a clear mprovement on the usual stuaton of havng to select and defend a sngle model, but t s of course not a panacea. A researcher never knows whether all relevant models have been ncluded and, although ts results are more robust than sngle-model approaches, t s always possble to come up wth a dfferent lst of models and produce a dfferent result. And so n 15
the end, and as always, the nvestgator s judgment always plays an mportant role n makng nferences. Model averagng cannot substtute for judgment, but t can help account for model uncertantes where pror knowledge s not avalable. Furthermore, n the present case, where 100% confdence ntervals are avalable (n the form of bounds on the parameters), we also have addtonal constrants on possble results. A Techncal Issues n Modelng and Estmaton From one applcaton of one specfcaton of ths model, we compute the posteror densty of a quantty of nterest by drawng t from ts posteror, condtonal on the model P( M k, T ). To do ths, we draw smulatons of β bad and β good from ther posteror and calculate smulatons of. Bayes theorem specfes that the posteror s proportonal to the product of the pror tmes the lkelhood, P(Θ T ) P(Θ)P(T Θ), where P(Θ) s the pror probablty dstrbuton on some unknown parameter Θ, and P(T Θ) s the lkelhood. Everythng s condtoned on X, N, and Z, whch we observe. We use the standard ndependent pror on each parameter of Θ as descrbed n Kng (1997). Ths pror dstrbuton and the lkelhood functon together defne Kng s model n a standard Bayesan framework. Let M k denote the kth model specfcaton (k = 1,..., 31). Then we make an nference about a quantty of nterest by computng ts posteror dstrbuton va Bayesan model averagng. To do ths, we frst compute, for each model, the posteror dstrbuton of (computed from the posteror wth beng some known functon of Θ): P( M k, T ). Then we average over these models by weghtng by the relatve posteror probablty that each model s correct gven the data, Pr(M k T ): Pr( T ) = 31 k=1 P( M k, T ) Pr(M k T ), where the posteror model probablty s Pr(M k T ) = P(T M k ) Pr(M k )/[ 31 j=1 Pr(T M j) Pr(M j )]. Ths s the probablty that model k s correct, gven the set of models n the analyss; t should not be confused wth R 2 -lke measures whch typcally reward models that over-ft wthout dstngushng systematc from dosyncratc features of the data. To compute the posteror model probablty, we need two elements. Frst s a pror probablty that each model s correct, Pr(M k ), whch we set to unform. The other s the margnal lkelhood, P(T M k ), whch s obtaned by averagng the lkelhood over the pror dstrbuton. 12 The margnal lkelhood s the probablty of seeng the data that actually were observed, calculated before any data became avalable (Kass and Raftery, 1995, p.776). That s, nstead of maxmzng the lkelhood wth respect to the parameter gven the data, as we would do to compute the maxmum lkelhood estmate, the margnal lkelhood does not have a maxmzaton step: t s the average value of the lkelhood evaluated at parameter values drawn from ther pror densty. (Although ths quantty could be computed by smulaton n ths way, such a method tends to be hghly neffcent, especally for problems wth relatvely flat prors or hgh dmensonal parameter vectors.) To compute the margnal lkelhood, we use the Laplace approxmaton, whch s known to perform well 12 The margnal lkelhood s P(T M k ) = R P(T Θ k, M k )P(Θ k M k )dθ k, where P(Θ j M j) s the pror dstrbuton for the parameter vector Θ n Model k. 16