What follows is, to the best of my understanding, largely folklore. Variants of the argument appear implicitly across the literature on Bayesian updating, relative entropy, and consistent inference, and the conclusion is often taken for granted once one is familiar with Jeffrey updating. However, I have not been able to find a place where the reasoning is laid out explicitly and in a self-contained way, starting from minimal assumptions and making clear what is, and is not, being imposed. For that reason, I am recording this note here, both for my own reference and in the hope that it may be useful to others. A PDF version containing the same material, with slightly improved formatting, is available here.

Let $X$ and $Y$ be finite-valued quantities (random variables) about which we wish to reason. Our prior information is summarized by a joint distribution

where $p(x)$ represents our prior state of knowledge about $X$, and $\varphi(y|x)$ encodes the likelihood. No interpretation beyond this bookkeeping role is assumed.

From $P$ we may compute the implied marginal distribution $P_Y(y)=\sum_x p(x)\,\varphi(y|x)$ and, by the product rule (I wouldn’t call this “Bayes’ theorem” yet), the inverse conditional distribution

Now suppose that new information becomes available which does not specify a particular value of $Y$, but instead constrains our revised beliefs about $Y$ to take the form of a probability distribution $\tau(y)$. The problem is then to determine what joint distribution $R(x,y)$ should represent our new state of knowledge, given that it must be consistent with $\tau$ and must not contain any information not logically implied by the prior $P$ together with this new constraint.

Principle of minimum change

Following the general principle that probabilities should be updated only to the extent required by new information, we select $R$ to minimize the relative entropy

subject to the constraint $R_Y=\tau$. This criterion ensures that no unwarranted assumptions are introduced.

Solution

Any admissible $R$ may be written as

Substitution into the relative entropy yields the identity

(Apologies for the ugly formatting; I’m afraid this is a bug of WordPress; remember that a better formatted version can be downloaded from this link.) The first term depends only on the revised marginal $\tau$ and is therefore fixed. The second term is nonnegative and vanishes if and only if

Hence the unique distribution consistent with the stated constraints and the principle of minimum updating is $R(x,y)=\tau(y)\,\hat\varphi(x|y)$, wherever $\tau(y)>0$. Note that, if it happens that $\tau(y)>0$ but $P_Y(y)=0$, we are in the situation in which the new evidence is falsifying our prior. This is signaled by the fact that, in this case, $D(\tau\|P_Y)=+\infty$.

Interpretation

The new information alters only our beliefs about $Y$; therefore, rational consistency requires that our conditional beliefs about $X$ given $Y$ remain exactly those implied by the prior. Any other choice would amount to smuggling in additional information not contained in the premises.