The Pursuit of Objec1vity in Science and Public Life

Transcription

1 The Pursuit of Objec1vity in Science and Public Life Theodore M. Porter Force From Numbers. What do we trust? 1

2 Aliis exterendum. In the 1830s,the Sta1s1cal Society of London chose as its moho: Aliis exterendum, to be threshed out by others The society briefly called for facts only, prohibi1ng expressions of opinion. The sheath survives, though the slogan is gone 2

3 3

4 A famous and conten1ous argument holds that computers with access to Big Data can do without models or conceptualiza1on. Does respect for data imply intellectual passivity, following rules? Is that objec&vity? The prac5cal work of sta5s5cs is not simple. 4

5 These authors give a history of objec1vity as a sequence of mostly incompa1ble concep1ons: Truth to nature (ca to 1840) Objec1vity, or mechanical objec1vity (ca to 1910) Professional judgment (20 th century) Was impersonal, rule- bound objec1vity at it peak in the late 19 th century? Maybe it has never been more powerful than now. 5

6 One of the more compelling expressions of the mechanical ideal of quan1ta1ve objec1vity was cost- benefit analysis, which grew up ca to 1960 in the United States. It arose from conflicts about river control for canals, irriga1on, and electric power genera1on. Private companies opposed state- sponsored compe11on. There were in addi1on strong rival agencies. The logic of measurement here was elusive. Maybe it was more Important for decisions to be well standardized. Pine Flat Dam on the King s River in California, object of a biher dispute between the Corps of Engineers and the Bureau of Reclama1on. 6

7 Early computa1on had some role in the press for rou1nes of objec1vity, but perhaps as effect as much as cause. The algorithm, a keyword of the early postwar Period, is a model of mechanical reason. The military was a great sponsor of algorithms. John von Neumann 7

8 In science, the sta1s1cal test of significance was a great missionary for rou1nes of objec1vity. Was this the whole purpose of an experiment, to give the data a chance to reject the null hypothesis? Is sta5s5cal significance really the main thing? 8

9 Sta1s1cs, as its name indicates, was originally about poli1cal or state numbers. This took in poli1cal economy. Sta1s1cal mathema1cs grew up around gene1cs and agriculture as well as social and economic numbers. Medicine had a role from the beginning. In recent decades it has become central. Yet the medical profession was always a bit reluctant. And even when it is eager, where do doctors get their evidence? Oien from interested par1es. And when not, oien in a form that is needlessly difficult, so they don t understand. 9

10 The randomized controlled trial has a compelling logic. Randomiza1on provides reasonable assurance that the group receiving the new treatment is comparable to the control. It also creates the condi1ons for an error calcula1on. But 10

11 It is not possible to rou1nize treatment fully. What about pa1ents who withdraw from the experiment? What is the proper alterna1ve treatment? A placebo? The currently accepted treatment? Do you approve a less effec1ve drug on the basis of differen1al side effects? Perhaps some pa1ents don t tolerate the best drugs. Now medical discre1on is back, and hard to control. Doctors may receive benefits for prescribing drugs that remain on patent. 11

12 A gold standard invites counterfei1ng Bad money drives out good. Also in randomized trials, unless we are vigilant. 12

13 Are numbers deceilul in a way that words are not? Indeed yes. 13

14 Are numbers deceilul in a way that words are not? Indeed yes. They are not more deceilul, but they can be decep1ve in different ways. Especially when they move beyond descrip1ve measurement to func1on as targets or goals, a basis for rewards. 14

15 Over the years, many sta1s1cians have proposed to use numbers and sta1s1cs as a neutral measure of the quality or effec1veness of government. Rousseau once said that you can tell a good government very simply, by whether the popula1on increases. In out 1me, the chosen measure is likely oien GDP. 15

16 It had a certain validity in a 1me of terrible economic depression. It worked rather well for reorganizing economies during World War II. It has been much cri1cized as a measure in recent years. It deals badly with unpaid labor, notably household labor. Environmental values are hard to measure, and discoun1ng prac1ces deal harshly with the long- term future.

17 (Donald) Campbell's Law: "The more any quan1ta1ve social indicator is used for social decision- making, the more subject it will be to corrup1on pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." (Also Charles Goodhart: When a measure becomes a target it ceases to be a good measure.) 17

18 The soi power of numbers and indicators is characteris1c of our 1me. One focus of discontent is the PISA process (Program for Interna1onal Student Assessment). It creates an irresis1ble regime of tes1ng. These tests cannot be merely measures. They define new structures of incen1ves.

19 Numerical comparison can be valuable. However BeHer numbers do not always correspond with genuine improvement. 2. Different poli1cal, cultural, and economic systems may have good reasons to pursue different goals in differently- organized systems. 19

20 Quetelet declared (in La1n!): Mundum numeri regunt - numbers rule the world. Power from Numbers yes, but the data don t interpret themselves. We should beware of trea1ng sta1s1cs as a technical field, in which sta1s1cians and subject specialists (guided, perhaps, by poli1cal leaders), seek the best measures even when they do not correspond to the best outcomes..

21 We should no1ce that the power of numbers and indicators may not require state power. Here, ra1ngs produced privately by an American magazine reshaped the process of law school choice, and even the law schools themselves. The pres1ge hierarchy it defined, though harshly cri1cized, became real because people acted on it. Here numbers are like words in the books we write. Once they go out the door, the author has lihle influence on what they will be taken to mean. 21

22 If you hope to use indicators and other measures to make the world navigable in simpler terms Be careful what you wish for. 22

23 A few lessons. If objec1vity is supposed to be something mechanical, standing above interpreta1on, it may not be what we really want. Idealizing the bohom line, a single measure, can create a lot trouble. Plural measures have some definite advantages. Perhaps it is beher to recognize the crucial role of trained, careful, commihed individuals in the work of science and sta1s1cs, rather than to idealize an impossible subjectless objec1vity. We cannot dispense with expert interpreta1on, all the more so because cannot in general be precisely and strictly defined. 23

24 And finally The public mahers. Many public numbers are intrinsically value- laden, at least in the sense that they have consequences for people A public measurement is oien already an ac1on. It is necessary to consider in advance how people will react to it. 24