Pattern recognition applied to presidential elections in the United States, : Role of integral social, economic, and political traits

Transcription

1 Proc. Natl Acad. Sc. USA Vol. 78, No., pp , November 98 Poltcal Scence Pattern recognton appled to presdental electons n the Unted States, 86-98: Role of ntegral socal, economc, and poltcal trats (algorthm of recognton/parametrzaton of qualtatve problem/ dynamcs of socety) A. J. LICHTMAN* AND V.I. KEILIS-BOROKt *Department of Hstory, AI\Iercan Unversty, Washngton, D.C. 26; and tnsttute of the Physcs of the Earth, Academy of Scences, Moscow, U.S.S.R., Contrbuted by V. I. Kels-Borok, August, 98 ABSTRACf 'fle outcome of Amercan presdental electons n follows certan regular patterns whch can be descrbed phenomenologcally by smple ntegral parameters of "common sense" type. Despte a vast lterature on voters and electons, there s no adequate model of the process that leads to vctory for the wnners n Amercan presdental electons. In leu of such a model, we use pattern recognton to dsclose emprcal regulartes that may contrbute to a better understandng of the electoral process. We hypothesze that the outcome of electons follows the dynamcs of certan smple, ntegral parameters that depct socal, economc, and poltcal crcumstance. We show that two dstnct types of stuatons dagnosed by these parameters preceded electons won by ncumbent and by challengng partes, respectvely. In the statstcal test of our conclusons we reject the smplest competng hypothess-that the outcome of an electon s ndependent of our dagnoss of the stuaton. We nether clam that other parameters cannot be used for the same purpose nor suggest methods for predctng future electons. Tradtonal approaches to presdental electons emphasze ether the party dentfcaton of voters or the mpact of partcular ssues. Yet party loyalty remans hghly stable over long perods of tme whereas the role of ssues s contnually changng. To understand the outcome of electons we need to examne socetal trats that are more dynamc than party dentfcaton and yet transcend the decsons made by ndvdual voters, the movement of voter blocs, the unque ssues of an electon, and the stratagems of campagners. If successful, our analyss would show that the process of retanng or rejectng an ncumbent party follows regular patterns ndependently of the turbulence of partcular campagns. Gven the small number of presdental electons, the relatvely large number of potentally nformatve trats, and the absence of gudng theory, pattern recognton s an approprate procedure for ths study. As a heurstc devce for orderng and condensng nformaton, pattern recognton can dsclose relatonshps that otherwse would be concealed from vew (-3). DATA 'fle objects of our analyss are Amercan presdental electons from 86 to 98. Electons, dentfed by ther year, are dvded nto two classes: I, the Incumbent party gans a pluralty of the popular vote (whether or not the ncumbent presdent was a canddate for reelecton); and C, the Challengng party gans a pluralty of the popular vote. For two electons, 876 and 888, the tally of electoral votes reversed the preference expressed by the popular vote. Therefore, we tabulated 876 as belongng to class C (reflectng the popular vote pluralty for challengng party canddate Samuel 723 J. Tlden rather than the electoral vote vctory for ncumbent party canddate Rutherford D. Hayes) and we tabulated 888 as belongng to class I (reflectng the popular vote pluralty for ncumbent party canddate Grover Cleveland rather than the electoral vote vctory for challengng party canddate Benjamn Harrson). In classfyng the electons of 88 and 892, the party that actually ganed the presdency was consdered the ncumbent party. Each electon year s descrbed by a set of answers to a questonnare (Table ). The questons are the same for each electon year, but the answers vary accordng to the crcumstances prevalng n gven years. The questonnare can be answered pror to the comng electon; most questons can be answered defntvely by the tme both major partes have selected ther nomnees. The answers to some questons may be altered durng the course of a campagn-for example, the occurrence of socal unrest or scandal. All questons are to be answered at the lowest possble level of resoluton: "yes" or "no." The answer to each queston reflects dverse features of complex stuatons that may be both causes and symptoms of the prospects for ncumbent and challengng partes. Socal unrest, for nstance, may ndcate dssatsfacton wth the status quo whle tself becomng a reason for rejectng the ncumbent party; the ncumbent party n tum may attempt to explot dsorder to dscredt the opposton and rally supporters. Table 2 dsplays our data set, dsclosng the answers to all2 questons for each electon year. Answers to several of the questons, notably nos. 8-2, depend on judgments about hstorcal crcumstance that are frequently put forth by scholars but are not reduced to precse numercal crtera. By offerng our full set of data, we enable readers to make judgments of ther own about how to answer these questons and to compare the results obtaned from the use of alternatve responses wth the fndngs we present. ALGORITHM The data n Table 2 consttute our "learnng materal." As always n pattern recognton, our problem s to derve from ths materal a "rule of recognton" that can assgn an electon to class I or C, gven the answers to the questonnare for that partcular electon. If found, such a rule would dsclose the "dstnctve trats" that dstngush electons won by the ncumbent party from those won by the challengng party. Because the learnng materal s lmted (8 electons of class I and 3 electons of class C), we sought to choose as smple an algorthm as possble for the dervaton of a rule of recognton. We have chosen the algorthm "Chemng's dstance" suggested n ref. 4; a bref descrpton of t follows. Each electon year s descrbed by the bnary vector Y ;(X,X 2,..., X"), where X; = or X; = represents the answer to the

2 Poltcal Scence: Uchtman and Kels-Borok Proc. Natl Acad. Sc. USA 78 (98) 723 Table. Questonnare. Has the ncumbent party been n offce more than a sngle term? (no) 2. Dd the ncumbent party gan more than 5% of the vote cast n the prevous electon?* (yes) 3. Was there uujor thrd party actvty durng the electon year? (no) 4. Was there a serous contest for the nomnaton of the ncumbent party canddate? (no) 5. Was the ncumbent party canddate the sttng presdent? (yes) 6. Was the electon year a tme of recesson or depresson? (no) 7. Was the yearly mean per capta rate of growth n real gross natonal product durng the ncumbent admnstraton equal to or greater than the mean rate n the prevous 8 years and equal to or greater than %?t (yes) 8. Dd the ncumbent presdent ntate uujor changes n natonal polcy? (yes) 9. Was there uujor socal unrest n the naton durng the ncumbent admnstraton? (no). Was the ncumbent admnstraton tanted by uujor scandal? (no). Is the ncumbent party canddate charsmatc or a natonal hero? (yes) 2. Is the challengng party canddate charsmatc or a natonal hero? (no) The answers n parentheses favor the vctory of the ncumbent party accordng to analyss of the whole data set n Table 2 (last kernel n Table 3) Rounded to the nearest percent. t Pror to the 89s, the avalable statstcs are approxmate. th queston n the questonnare. For each queston, two numbers are computed that ndcate the predomnant values of X; n class I and class C: P(/I) = n(,i)/n(i), and P(/C) = n(,c)/n(c). Here, n(,i) s the number of electons n whch X; = for class I, n(,c) s the number of electons n whch X;= for class C, and n(i) and n(c) show how many electons I and Care ncluded n the learnng materal. These ratos are then used to form a "kernel" representng the dstnctve trats (.e., the set of preferental answers for vctory by an ncumbent canddate). The kernel s a bnary vector (KI>J4.,..., K,J. K; = f P(/ I) - P(/ C) ;::: k; K; = f P(/ C) - P(/I);::: k; otherwse, X; s not used n the kernel. Table 2. Answers to questons n Table ( =yes, =no) Electon Qu::. est_ o_n_n...:umbe=:..::..::...r.::n:..t.:..a.:..b:.::le:...:.. year « W Ineumbent vctory Challencer vctores P(/CJ P(/ll The dstance D between the kernel and a gven electon s defned, as D = W;«'>(K;,XJ. j Here «'>(K;,XJ = f K; =F X; (showng that the value of X; for that electon year dffers from the value assocated wth vctory for the ncumbent) and «'>{K;,XJ = when K; =X;. We assume here that W; =, assgnng equal weghts to the answers to each queston. Then D becomes the number of answers preferental for class C; t s Chemng' s dstance, and hence the name of the algorthm. Let us denote DI, the maxmal value of D for all precedng electons I, and DC, the mnmal value ofd for all precedng electons C. We wll recognze an electon as I f D < DC and D :s DI and as C fd > DI andd DC. H nether of these condtons s satsfed, the recognton s ndefnte. :: We also assumed k =.. ANALYSIS The last two lnes of Table 2 show the values of P(/) and P(/ C) for all the learnng materal. The correspondng kernel s dsplayed n the last lne of Table 3. Table 4 gves the value of D for each electon. Several aspects of these results mert dscusson. () Taken ndvdually, the answers to none of the questons can accurately dvde electons nto those won by ncumbent and by challengng partes. Ths can be seen from Table 2. One queston (Was there a serous contest for the nomnaton of the ncumbent party) emerges as by far the most powerful dscrmnator. The answer to ths queston dentfes ncorrectly only one of the ncumbent vctores and three of the challenger vctores. The specal relevance of ths queston suggests that nomnaton contests wthn the ncumbent party are both sympto- *We also used the algorthm CORA-3 (-3) to generate, from the learnng materal n Table 2, multple dstnctve trats, representng combnatons of answers to several questons. Such trats can provde addtonal nsght nto the nature of the dfference between electons I and C, but wth lmted learnng materal the multple trats are especally susceptble to the nfluence of random, spurous combnatons of the answers. Multple trats, however, showed no sgnfcant addtons to the results generated by the algorthm "Chemng' s dstance. " Wth a broader set of questons, multple dstnctve trats may be more revealng.

3 7232 Poltcal Scence: Lchtman and Kels-Borok Table 3. Kernels based on nformaton accumulated through consecutve electon from 86 to T Year Queston number T K = f P</[) - P(/C).; K = o f P(/Cl - P(j[).; K = * otherwse. t Ths kernel s repeated n Table, as the answers gven n parenthess. matc of a lack of confdence n the ncumbent admnstraton (especally when an ncumbent presdent s seekng renomnaton) and productve of potentally damagng dsunty wthn the rulng party. Snce 86, only one ncumbent party canddate-james A. Garfeld n 88-survved a serous contest for the presdental nomnaton. Taken together, the answers to our questonnare dvde the electons nto classes I and C as shown n Table 4. Electons of class I have dstance D :5 5 and electons of class C have dstance D :::::: 5. Three electons attan the ndetermnant dstance of D = 5; two of them belong to class I and one, the electon of 92, belongs to class C. Ths overlap has a clear explanaton whch s offered not to dsspell an nconvenent result but to show that certan factors may be of suffcent weght n partcular electons to overrde other consderatons. The presdental electon of 92 s consdered to be one of the most unusual n the naton's hstory. Dssatsfed wth the polces of hs hand-pcked successor Wllam Howard Taft, former Republcan presdent Theodore Roosevelt sought to wrest the 92 nomnaton from Taft and agan become the Republcan contender. Falng n ths effort, he organzed a thrd-party movement that splt the Republcan vote and handed the electon to Democratc nomnee Table 4. Dstances for presdental electons computed from the kernel for (last lne n Table 3) C C 98C / 92C 896C 92C C 96C 876C C 932C 86C 952C DstanceD Proc. NatL Acad. Sc. USA 78 (98) Table 5. Test on mutually ndependent data Kernel and DC,5lfor 86o-92 DC=3;5l= DC= 6;5 = 4 Electons of Indetermnate (928/, 932C, 96C, 976C) (98, 88, 92C) Identfcatons Wrong Correct 2 2 Frst kernel (86-92) s gven n the lne 92 of Table 3; second kernel (86-92) s the same as n the last lne of Table 3. The only dfference between these kernels s that n the fll'llt kernel, three trats (nos. 7,, and 2) are elmnated. Woodrow Wlson. Together, Taft and Roosevelt polled 5% of the vote, compared wth 42% for Wlson. () The defnton of the kernel may predetermne and enhance the separaton of electons I and C on the axs D (bootstrap effect). The smplest hypothess, whch may compete wth our conclusons, s that our recognton s rrelevant to the outcome of electons. To test ths hypothess, we ncluded n the learnng materal only the 6 electons from 86 to 92 and determned the correspondng kernel. Then we used ths kernel to dentfy the 5 electons from 924 to 98. Also we reversed the procedure: determned the kernel for. the electons of and used ths kernel to dentfy the electons of The results of ths experment are shown n Table 5. They are satsfactory: the competng hypothess can be rejected at the level above 99.9%. A more refned competng hypothess would be to assume that the probabltes of electons I and C are related, as n(i):n(c) (9:7 n 86-92; 9:6 n ). Ths hypothess s also rejected at the level above 99.9%. As an addtonal test, we compared the dfferences P(/ I)-P(/ C) for the two halves of our tme perod (86-92 and ). For all tems n the questonnare, the sgn of ths dfference for s the same as for Accordngly, the competng hypothess-that the learnng materal s a realzaton of a random bnomal process, ndependent of the outcome of electon-s rejected at the level above 99.9%. () The kernel for dscrmnatng electons of classes I and C proved to be hghly stable throughout the tme perod covered by our analyss. Ths s clear from Table 3 whch dsplays the consecutve kernels created frst by usng learnng materal only from 86 and 864 and then addng learnng materal from each succeedng electon, one by one, untll98 s reached. Table 3 reveals not a sngle change after 88 n the values of the dstnctve trats. The only change from year to year s that dfferent questons are elmnated. No queston, however, s elmnated after 952. The fnal values of the kernel confrm what mght be expected from a knowledge of Amercan poltcs. The full set of favorable crcumstances for contnuty n party control of the Whte House has not changed sgnfacantly n the past years. (v) To test further the stablty of our recognton rule, we performed an experment smlar to the experment "earthquake's hstory" descrbed n ref. 3. Begnnng wth 896, we successvely used each of the kernels dsplayed n Table 3 to compute the dstrbuton of D for all electons ncluded nto the learnng materal for the formaton of the kernel. We then used the same kernel to determne the value of D for the upcomng electon (not ncluded n learnng materal) and to project ts outcome. Ths experment smulates the stuaton of an observer Uchtman, A. J. and Lord, J. B., III (979) Party Loyalty and Progressve Poltcs: Quanttatve Analyss of the Vote for Presdent n 92, Annual Conventon, Organzaton of Amercan Hstorans, New York, Aprll979.

4 Poltcal Scence: Lchtman and Kels-Borok Proc. Natl Acad. Sc. USA 78 (98) 7233 Table 6. Separaton of electons I and C and projectons of upcomng electons (based on data from 86 to T, nclusve) Incumbent wns Actual outcome* Challenger wns Projecton for T Dstances Dstances next c I I (correct) c I I (correct) c I I (correct) 92 c I I (wrong) c I I (correct) 92 c I C (correct) c I I (correct) c I Indetermnate 932 c I C (correct) c I I (correct) c I I (correct) c I I (correct) c I I (correct) 952 c I C (correct) c I I (correct) 96 c I C (correct).

5 7234 Poltcal Scence: Lchtman and Kels-Borok Proc. Natl Acad. Sc. USA 78 (98) Table 6. Contnued c I I (correct) 968 c I C (correct) c I I (correct) 976 c I C (correct) 98 c I C (correct) Numerals!t to are dstances D. *Values of D correspond to kernels computed for nterval 86 tot. The numbers n the table under a dstance scale ndcate how many electons attaned that dstance; upper rows correspond to electons C, and lower rows, to electons I. Arrow ndcates the value of D for the upcomng electon (not ncluded n the learnng materal). Its year s ndcated above the arrow. who performs such pattern recognton before each presdental electon. Table 6 shows the results of ths experment-.e., the value of D for all electons of class and class C from 86 through the last electon ncluded n the computaton of the kernel. The frst column of Table 6 ndcates the year of ths last electon. The arrow ndcates the value of D for an upcomng electon; ts year s lsted above the arrow. For example, the kernel computed wth learnng materal from 86 to 94, generates values of D that range from to 5 for electons of class I and from 5 to 8 for electons of class C. The upcomng electon (944) has D =. Accordng to the rule, formulated above, vctory for the ncumbent party s projected.for 944. Vctory for the challengng party would be projected only n the case D > 5; for D = 5 the projecton would be ndetermnate. To facltate a vsual nspecton of the results we shfted to the rght the data from all cases n whch the challenger prevaled n the upcomng electon. The last column ndcates the projecton made accordng to the rule formulated above. Ths experment shows that n 9 of 2 cases the procedure correctly projects the actual outcome of a forthcomng electon. In one case--once agan the electon of 92-the projecton s ncorrect, forecastng an ncumbent vctory when n fact the challenger was to preval. In one case, 98, the projecton s ndetermnate. Thus, the dstnctve trats based on our questonnare proved to be hghly stable n ther ablty to classfy electons throughout the 2th century. Ths demonstraton of hstorcal stablty should not, however, be nterpreted as an evaluaton of a capacty to predct future electons. (v) We vared the questonnare and learnng materal to test the stablty of our results. The projectons of upcomng electons (Table 6) are relatvely most senstve to such varatons and we shall descrbe below the change n these projectons. Frst, we successvely removed each of the 2 questons from the analyss. Ths produced up to fve ndetermnate projectons, nstead of one, and up to two wrong projectons, nstead of one. Our 2 questons seem to be close to a mnmally necessary set, gven the nformaton ncluded n ths study. Of course, our questonnare could be superseded by other questons not consdered here. Second, we added fve tems to the questonnare-whether the ncumbent canddate was a Republcan or Democrat; whether there was a serous contest for the challengng party nomnaton; whether the electon occurred durng wartme; whether foregn polcy ssues were promnent durng the campagn; and whether domestc ssues were promnent. For the largest part of the 2th century these parameters dd not pass the crtera for ncluson n the kernel. When ncluded, they changed the projectons for 96, 968, and 976 from correct to ndetermnate. Thrd, we removed from the learnng materal fve electons-88, 884, 888, 96, and 968-wth less than % dfference n the proporton of the popular vote garnered by ncumbent and challengng partes. Ths changed two projectons: for 924 and 932, from correct to ndetermnate. Fourth, we excluded all 9th century electons, formng our fnal kernel only from the electons of Ths procedure altered not a sngle component of the kernel. These experments llustrate a hgh stablty of the recognton rule. Fnally, we should note that our choce of k =. for the selecton of trats s not based on a statstcal model of the value P(/I)- P(/C). Actually, ths value s>.2 for all trats, and our results are stable to varaton of k. Ths work was accomplshed whle both authors were Farchld Scholars at The Calforna Insttute of Technology. Dr. A. Raefsky and Dr. E. Nyland wrote the programs we used. Ths paper s Contrbuton 367, Dvson of Geologcal and Planetary Scences, Calforna Insttute oftechnology, Pasadena, CA Bongard, M. (968) Raspoznavane obrazov (Pattern recognton), Nauka, Moscow, n Russan. 2. Press, Frank & P. Brggs (975) Nature (London), 256, Gelfand, I. M., Sh. A. Guberman, V.I. Kels-Borok, L. Knopoff, F. Press, E. Ya. Ranzman, I. M. Rotwan & A. M. Sadovsky (976) Phys. Earth Planet. Inter., Gvshan, A. D., A. V. Zelevnsky, V. I. Kels-Borok, & V. I. Kosobokov (98) n Methods and Algorthms for Interpretaton of Sesmologcal Data, Computatonal Sesmology V. 3, Nauka, Moscow, n Russan.