Today I taught Bayes’ rule and thought I would collect everything in a blog post. This post was written with the help of an LLM, based on a transcript of the class recording and the ramblings that continued later in my office.

Bayes’ rule is often introduced through elementary examples involving urns filled with colored balls. We know the composition of each urn, we draw a ball, and we ask how the observation changes the probability that the ball came from one urn rather than another.

In this setting, everything is straightforward. The experiment is specified in advance, all relevant probabilities can be interpreted as fractions of possible outcomes, and Bayes’ rule follows from the elementary laws of probability. The observation tells us something about the urn, but does not require us to reconsider the statistical model describing the experiment.

However, when we move from urns to the much more general problem of updating our beliefs in light of new evidence, something conceptually important happens. We are no longer simply calculating conditional probabilities within a fixed statistical model. We are prescribing how an observer should change their beliefs when new information becomes available.

These two problems look almost identical mathematically, but they are logically different.

Two statements that look the same

Let denote a hypothesis and an observation. Suppose that, before the observation, our beliefs are represented by a probability distribution .

The usual formula reads

This is a mathematical identity. It follows directly from the definition of conditional probability, with the denominator computed, when necessary, using the law of total probability.

Notice, however, that all probabilities appearing in this formula belong to the same probability distribution, .

Now suppose that we actually observe , and denote by our new state of belief. The Bayesian prescription is

This is a different statement.

The first equation tells us what the conditional probability of given was before the observation. The second tells us what our probability for should become after the observation.

Nothing in the elementary laws of probability requires these two quantities to coincide. Indeed, the laws of probability constrain a probability distribution at a given time, but do not, by themselves, prescribe how one probability distribution must evolve into another.

The step from the first equation (Bayes’ rule) to the second (Bayesian conditioning) requires an additional principle of rational inference.

Ramsey had already noticed this in 1926

This distinction had been formulated very explicitly already a century ago by Frank P. Ramsey, one of the founders of the subjective interpretation of probability.

In his celebrated essay Truth and Probability, written in 1926 and published posthumously in 1931, Ramsey carefully distinguishes the degree of belief in a proposition conditional on from the degree of belief that an individual would actually have in after learning :

“knowledge of q might for psychological reasons profoundly alter his whole system of beliefs.”

This is from page 180 of the original publication, which can be found in this electronic edition of Ramsey’s essay.

Ramsey was not suggesting that every psychological change of belief is rational. Rather, he was pointing out that conditional belief and actual belief revision are conceptually different things.

The question is therefore whether, and under what conditions, the identification of such changes is rationally justified.

When is Bayesian conditioning justified?

This question has an extensive history in the philosophy of probability, where the updating prescription is usually called the Principle of Conditionalization.

Richard Jeffrey, in Conditioning, Kinematics, and Exchangeability (1988), identifies two conditions under which ordinary conditioning is appropriate.

The first is certainty: after the observation, the event is regarded as certain,

The second is sufficiency: learning does not change our conditional beliefs given ,

Together, these conditions imply the usual Bayesian updating rule.

The crucial point is that the second condition is not imposed by probability theory. It is an assumption about what the observation has, and has not, changed in our state of knowledge.

Jeffrey also developed a more general theory of updating, now known as Jeffrey conditionalization, which allows an observation to change probabilities without making any particular event certain. This theory was subsequently investigated in detail by Persi Diaconis and Sandy Zabell in their paper Updating Subjective Probability (1982) and by Richard Bradley, in Radical Probabilism and Bayesian Conditioning (2005).

Why should we follow Bayes’ rule, then?

Once we recognize that conditioning is an additional rule, rather than a mathematical identity, the natural question is how to justify it.

Several answers have been proposed.

One influential approach is based on consistency across time. David Lewis, in Why Conditionalize? (written in 1972 and published in 1999), develops a diachronic Dutch-book argument. Under appropriate assumptions about an agent’s betting dispositions and updating policy, deviating from conditionalization can expose the agent to a sequence of individually acceptable bets that together guarantee a loss.

Another approach is based on epistemic utility. In Justifying Conditionalization: Conditionalization Maximizes Expected Epistemic Utility (2006), Hilary Greaves and David Wallace show that, under specified conditions on epistemic utility, conditionalization is the unique updating rule that maximizes expected epistemic utility.

A third possibility is to invoke a principle of minimum change: when new information arrives, our beliefs should change only to the extent required to accommodate that information, and not more.

For example, suppose that is our prior distribution and that the only new constraint is that an event has become certain. Among all distributions satisfying , we may select the one minimizing the relative entropy

For , this procedure uniquely selects

namely, ordinary Bayesian conditioning.

The argument extends naturally to Jeffrey updating when the new information specifies an updated marginal distribution rather than a certain event. I have discussed this construction in a separate note on this blog.

All these approaches have something important in common. They do not derive the Bayesian updating prescription from the elementary laws of probability alone. Instead, they identify additional requirements that select conditionalization as the appropriate rule.

Should we care?

From my viewpoint, this problem becomes particularly important when we attempt to generalize Bayesian reasoning beyond classical probability theory, as in quantum Bayesian retrodiction, where we do not even have a Bayes’ rule to start from, since any attempt to model events as balls in urns encounters a fundamental no-go.

This suggests that the starting point are the laws of consistent reasoning and that Bayes’ rule should follow from those, not vice versa.

---

References and further reading

• F. P. Ramsey (1926), Truth and Probability, published in The Foundations of Mathematics and Other Logical Essays (1931), pp. 156–198. See especially p. 180.
• R. Jeffrey (1988), Conditioning, Kinematics, and Exchangeability, reprinted in Probability and the Art of Judgment (1992).
• P. Diaconis and S. L. Zabell (1982), Updating Subjective Probability, Journal of the American Statistical Association 77, 822–830.
• D. Lewis (1972/1999), Why Conditionalize?, in Papers in Metaphysics and Epistemology, pp. 403–407.
• R. Bradley (2005), Radical Probabilism and Bayesian Conditioning, Philosophy of Science 72, 342–364.
• H. Greaves and D. Wallace (2006), Justifying Conditionalization: Conditionalization Maximizes Expected Epistemic Utility, Mind 115, 607–632.
• S. Wenmackers and J.-W. Romeijn (2016), New Theory about Old Evidence: A Framework for Open-Minded Bayesianism, Synthese 193, 1225–1250.
• Bayesian Epistemology, Stanford Encyclopedia of Philosophy, especially Sections 1.2 and 5, for an overview of conditionalization, its justifications, and its limitations.