DA VINCI DIDN’T PAINT BY NUMBERS:

MAYO’S CRITIQUE OF THE BAYESIAN WAY

Sam George

6/15/99

PROLOGUE:

On what epistemological grounds do we make scientific truth claims? In Error and the Growth of Experimental Knowledge (all page numbers without an author’s name preceding them refer to this text), Deborah Mayo claims the appropriate focus is on small-scale statistical inferences grounded in actual experimental practice (p. 59). Rather than exclusively seeking experimental results that support or cast doubt on a theory-laden supposition, Mayo asserts along with the New Experimentalists that “experimental knowledge remains despite theory change” (p. 62). This puts her in direct opposition to the proponents of the Bayesian Way. The various branches of Bayesianism strongly assert a theoretical presumption that prior degrees of belief are modified by the evidence to yield posterior degrees of belief. Mayo concedes that statistical testing approach results can be reconstructed after the fact by the Bayesian approach, yet one of her basic theses is that the Bayesian approach is not and cannot be the method used by scientists who seek the growth of experimental knowledge. Even though masterpieces of painting, such as the Mona Lisa, can be reproduced by children following a paint-by-the-numbers kit, this does not mean that the original artist did or even could make his or her original contributions through such a method.

ACT I: MAYO ON THE BAYESIANS

Mayo began her pointed critique of Bayesianism with a contrast between evidential-relationship and testing approaches.

E-R approaches . . . commonly seek quantitative measures of the bearing of evidence on hypotheses. What I call testing approaches, in contrast, focus on finding general methods or procedures of testing with certain good properties. (p. 72)

A major difference between the two is that E-R approaches generally (although not exclusively) assign probabilities to hypotheses, while testing approaches do not. Objective Bayesians in the tradition of Rudolph Carnap attempted to deduce the prior probabilities from the logical structure of some first-order language (p. 73). Unfortunately, the first-order languages used were inadequate to articulate actual science. The majority of Bayesians, on the other hand, subscribe to subjective degrees of belief for prior probabilities. Mayo quotes Howson and Urbach,

we are under no obligation to legislate concerning the method people adopt for assigning prior probabilities. These are supposed merely to characterise their beliefs subject to the sole constraint of consistency with the probability calculus (p. 75).

Strict Bayesians must accept that different persons will have radically dissimilar opinions, both between people and also for the same person from time to time. The Bayesian Way provides for them the only rational method of fitting together prior and posterior degrees of belief via Bayes’s theorem. The argument goes along the lines that “if we are rational, we will be coherent in the Bayesian sense” (p. 76), e.g. avoiding Dutch Book arguments, which would insure that the gambler would lose money. Mayo quoted the Bayesian Leonard Savage as writing “that the theory of personal probability ‘is a code of consistency for the person applying it, not a system of predictions about the world around him’” (p. 76).

E-R approaches ultimately lead to subjectivism, according to Mayo. While some things, such as binomial probabilities, can be calculated in an objective fashion, other probabilities — such as the composition of matter by atoms — can never be rigorously objective. Were we in the context of multiple universes, it would make sense to talk about a given hypothesis, H, having a nontrivial probability in a given universe. A person without complete knowledge, on the other hand, could meaningfully assign a subjective degree of belief between 0 and 1 for H even in a model of a singular universe.

Bayes’s Theorem can be generalized from probability theory.

For an exhaustive set of disjoint hypotheses, H1, H2. . ., Hn, whose probabilities are not zero, and outcome e where P(e) > 0:

P(e|H1)P(H1)

P(H1|e) = _________________________________________

P(e|H1)P(H1) + P(e|H2)P(H2) + . . . + P(e|Hn)P(Hn)

(pp. 81-82).

In certain well-defined cases, such as Mayo’s contrived second-order game of playing rouge et noir OR single-number roulette determined by the flip of a coin, the theorem works well and non-Bayesians agree with Bayesians on the appropriateness of the application. The rub comes, of course, when there are not well-defined and objective prior probabilities, as will be the case the vast majority of the time. The common person (along with the statistically-minded individual) will want to know how well a Bayesian model corresponds to reality; to the Bayesians, however, the issue is coherence, not correspondence.

Mayo pointed out that the Bayesians often defend themselves with the assertion that the differences in prior probabilities make no difference in posterior probabilities with sufficient additional evidence. This is not a credible defense of the Bayesian Way for Mayo for at least the reasons that (1) some agents could assign zero probabilities and different agents could disagree among themselves as to the assignments of other degrees of belief, and (2) the Bayesian Way claims to be a good (according to Howson and Urbach THE ONLY good) way for evaluating non-statistical as well as statistical hypotheses. Howson and Urbach apparently also claim the whole point of (Bayesian) inductive reasoning is inference from premise(s) to conclusion (p. 85). Analogously to deductive reasoning, the point is not the truth of the premises–that much to them is irrelevant–but the form of the inference. According to Mayo, this move seems to be an attempt to subsume induction under deduction, making an ampliative deduction, and as such, this Bayesian assertion seems immediately suspect at the very best.

This is the heart of the controversy, according to Mayo: a basic difference in the aims of the Bayesians from the various non-Bayesians. Agreeing to disagree with the Bayesians over the goals of an ampliative induction removes much of the controversy between the two groups, she claims. Mainstream statistical reasoning fails by Bayesian standards, as does the Bayesian Way fail by normative statistical standards. Is there room here for compromise? “Conceding the limited scope of the Bayesian algorithm might free the Bayesian to concede that additional methods are needed, if only to fill out the Bayesian account” (p. 86). Unfortunately, the strictest Bayesians do not appear likely to concede their major thesis that “All You Need is Bayes” to quote Mayo’s (p. 88) paraphrasing the Beatles. Without an imperialist perspective, the Bayesian Way becomes one tool among a set of many other tools (which in my judgment is how it ought to be–more on this at the end of the paper). Bayesians of the Howson and Urbach variety wish to demote mainstream scientists as being somehow covert Bayesian agents to the extent that they are producing any useful results. Of course, this can only be maintained by the most dogmatic ideologue–mainstream science can and does get along quite nicely without subjective Bayesianism (p. 89). “The Bayesian model is neater [than error statistics methods], but it does not fit the actual procedure of inquiry” (p. 99).

ACT II: GEORGE ON MAYO ON THE BAYESIANS

The first question in evaluating Mayo’s critique of the Bayesians is whether or not she has represented them accurately. Some important points of Mayo’s inadequacy should be addressed, including Bayesian extremism and the direction of movement from prior to posterior degrees of belief.

While the various flavors of Bayesians would take different positions on the issue, the plain vanilla ones–such as Howson and Urbach–are uncompromising absolutists–their way or no way, “Bayes or Bust” (p. 84). Knowing what I now know about statistically-based reasoning, this assertion borders on the absurd. Surely someone can be a rational agent without following the Bayesian Way or even a way that could be approximated by Bayesianism. Yet just because the Bayesians stake an extremist claim, is that warranted grounds to dismiss their reasoning? Certainly not, for some truths are radical truths not admitting of compromise. As unlikely as it seems, it may indeed be the case that it is indeed a matter of Bayes or bust, and an implied ridicule of such a position is not helpful.

Mayo quotes Kyburg as saying,

for any body of evidence there are prior probabilities in a hypothesis H that, while nonextreme, will result in the two scientists having posterior probabilities in H that differ by as much as one wants (p. 84f).

This point is technically true, but misleading. Starting off, any two scientists could have such different prior degrees of belief that the posterior probabilities can be as arbitrarily far apart as one would like to make them, but the important thing to note is the direction of the movement. With additional evidence, the posterior probabilities will converge on the actual value. The difference will indeed wash out, it is not a matter of movement away from the actual value.

Mayo’s basic thesis, that the Bayesian Way does not reflect scientific practice, is well-taken. We do well to recall “considerations of probability have played an important part in the history of science, [but] until very recently, explicit probabilistic arguments for the confirmation of various theories, or probabilistic analyses of data, have been great rarities in the history of science” (Papineau, p. 292). The analogy comes to mind with recent developments in Kosovo: after NATO airstrikes apparently forced Serbian forces out of Kosovo, the Russian military has moved in, claiming critical strategic positions. Likewise, after centuries of painstaking progress, mainstream scientists find a noisy bunch of Bayesians laying claim (in this case) to the entire field of enterprise.

I find Bayesian imperialist claims to be obnoxious and overstated. That statistically-based evaluation can be reproduced after the fact by Bayesian algorithms does not demonstrate that the Bayesian Way is even equal to, much less superior to, statistics. Indeed, the Mona Lisa can be reproduced by a child using a paint-by-the-numbers kit, but no artist worthy of the title paints in such a fashion.

Mayo claimed the Bayesian model was neater (p. 99), but it seems to me that keeping track of successive generation of prior and posterior probabilities would become incredibly cumbersome, even in an age of computers. The theory certainly has an elegant simplicity to it, but in practice, it would be far too complex to execute. Again, the theory does not fit accepted scientific practice.

EPILOGUE: GEORGE ON THE BAYESIANS

Most generally, are people inclined to be Bayesians? If this were the natural inclination of human beings in evaluating logical inferences, such would be a powerful (albeit nonconclusive) argument in favor of Bayesianism. Apparently most Bayesians believe as much, such as “what we ourselves take to be correct inductive reasoning is Bayesian in character that there should be observable and sometimes systematic deviations from Bayesian precepts” (Howson and Urback, p. 423). This strikes mf belief in what I have said, but it seems reasonable to propose that most of the putatively rational agents in my classes will have at least a significant degree of belief in my words, yet I always have some students who attend class erratically at best and still claim that they expect to pass, even when they have failed every assignment all semester long. Hence it seems that Bayesianism is not normative among scientists and is not the exclusive course of action of all putatively rational individuals in everyday life.

Bayesians are dogmatic imperialists. This seems most convenient for the Bayesians; it sounds analogous to asking a person of great faith if there is a God. “Of course there is,” might be the initial reply. “What about the presence of evil?” might be the next question, to which the next answer might be, “Evil only exists to sort the faithful from the non-faithful.” In short, the person’s belief system is non-falsifiable–there is no evidence one could bring to such a person that would change his or her beliefs. The opportunities for a strict Bayesian to admit of error also seem diminishingly small. However, despite the epistemic circle of coherence within which the Bayesians confine themselves, the rest of the world does want to know if something is true, not simply if it is self-consistent. I am always disturbed when ideology takes precedence over correct reasoning. In this regard, the orthodox Bayesians are as obnoxious as any group of religious fundamentalists, which is rather ironic given the subject matter of the Bayesian faith. Yet, as Mayo pointed out, Bayesians are not speaking in the same categories as concern the rest of us–they are making puddings, while statisticians and other error-conscious reasoners are making entrees of various kinds. The problem is that the Bayesians insist that there is no need of anything else, that pudding is by itself sufficient, and that simply cannot be the case. Pudding has its proper place, but it does not have the nutritional value to support a human being by itself, and any person who tried to subsist on pudding alone would rapidly succumb to the diseases of malnutrition. Likewise, a pseudo-science confined to Bayesian orthodoxy would not permit the kind of explosion of knowledge humanity has witnessed over the past few hundred years–it simply is not the way science is or should be done.

As I said earlier, that one makes extreme claims does not rule out the truth of one’s position. However, for an extraordinary claim that purports to account for all science, the Bayesian Way must offer extraordinary proof of its position, and I fail to see as much. I believe Bayesianism could and should be retained as one tool among many. Mayo implies as much through the discussion on pages 86-88. I can imagine situations wherein a Bayesian approach could give useful information, even without completely objective prior degrees of belief. I did not spend much time watching the trial of O.J. Simpson, but it seems to me that the prosecution could have profitably cited Bayes’s theorem. Certainly not ALL abused women who are murdered are murdered by their husbands/boyfriends–perhaps police statistics would indicate that only 75% of that population were killed by their significant others–certainly not a powerful enough argument by itself to satisfy the American legal system’s standards for determining guilt beyond a reasonable doubt. That statistic, though, could credibly serve as a Bayesian prior belief and interact with the evidence of the blood tests (which also have well-documented statistical probabilities) to yield a posterior degree of belief that would be highly unfavorable for the defense. Of course, such would not prove Mr. Simpson’s guilt in any deductively valid fashion, but it would make for a strongly persuasive argument.

Generally, it seems to me, Bayesianism is at its strongest when dealing with the subjective decisions of human beings. Mayo only alluded to this briefly. However, it seems good to me as well that we bring to the surface all of our implicit reasoning and avail ourselves of all the tools at our disposal when making important decisions, and in this more limited sense, the Bayesian Way is a useful tool among others.

CITED WORKS

Howson, Colin and Urbach, Peter. Scientific Reasoning: The Bayesian Approach,

second edition. Open Court: Chicago and La Salle, IL, 1989, 1993.

Mayo, Deborah G. Error and the Growth of Experimental Knowledge.

The University of Chicago Press: Chicago and London, 1996.

Papineau, David, ed. The Philosophy of Science. Oxford University Press: New York,

1996.