My world was shaken this morning when I read a proof that (1) “absence of evidence is evidence of absence.” I had always thought this maxim was: (2a) “absence of evidence is not evidence of absence;” or less strongly, (2b) “absence of evidence is not necessarily evidence of absence.” Here are some examples of statements I think are logically correct, and provide examples of (2a,b):
- The fact that we have not seen a tea pot orbiting the Earth (have we been looking?) does not falsify the claim that a tea pot is orbiting the Earth. (Surely there must be one on the Mir space station!)
- The fact that we have not received and decoded interstellar messages from extraterrestrials does not support the notion that extraterrestrials do not exist.
- We have yet to directly observe a Higgs boson. Thus, there are no Higgs bosons.
- The fact that we have not found weapons of mass destruction in Iraq does not necessarily mean there were no weapons of mass destruction in Iraq. (Have we searched all sand dunes? Maybe there is a futuristic bomb that we cannot detect. Have we searched in the past hour? Maybe some just crossed the border.)
Let’s look at how the author here attempts to demonstrate that (1) is true.
First he or she makes some definitions:
- Define \(H\) as some hypothesis claimed to be true.
- Thus, we call \(\sim H\) “absence”, i.e., the hypothesis is false.
- Let \(D\) be evidence of \(H\), which means \(P(H | D) > P(H | \sim D)\).
- Finally, we call \(\sim D\) “absence of evidence”.
So with these terms so defined, the maxim (1) becomes:
$$P(\sim H | \sim D) > P(H | \sim D).$$
That is, we should not believe the hypothesis given an absence of evidence.
Before we even start with the proof, I feel uneasy with four things. First, is the interpretation of \(\sim D\) as being “absence of evidence”. If \(D\) is evidence in support of \(H\), is \(\sim D\) evidence supporting \(\sim H\) (e.g., we have recorded many white swans and no black swans to support our claim “all swans are white”)? Or does \(\sim D\) mean we have yet to collect data that supports either \(H\) or \(\sim H\) (e.g., we have not even gone to the field yet, but we have a list of historical housing prices)? Second, we have not properly defined the probability measure for \(H\) and \(D\), and their joint probability measure so that we can justifiably use conditional probability. For instance, I have no idea of the sample space from which \(D\) comes, which is related to my first problem: if \(P(D) + P(\sim D) = 1\), then I would interpret \(\sim D\) as the collection of all the other data in the world, including baseball statistics from 1910. Third, I am concerned by the lack of definition of a posterior for \(D\) and \(\sim D\). The fact that we are to believe in \(H\) given the maximum likelihood rule is not so good if \(P(D)\) is extremely small (as I will show below). And if \(P(D) = 0\), then any conditioning on \(D\) will be meaningless. And finally, it seems a more proper definition of “evidence” is that if \(P(H|D) > P(\sim H|D)\), then \(D\) is evidence of \(H\).
Anyhow, starting from the author’s definition of evidence, the author proceeds
P(H | D) & > P(H | \sim D) \\
1 – P(\sim H | D) & > 1 – P(\sim H | \sim D)
by the fact that, e.g., \(P(H | D) = 1 – P(\sim H | D)\).
This means that
P(\sim H | D) & < P(\sim H | \sim D) \\
P(\sim H | \sim \sim D) & < P(\sim H | \sim D)
since \(D \equiv \sim \sim D\).
The author stops here, believing he or she is finished (and has even made it available for sale printed on a shirt. Oh the hubris!).
However, the author has only shown that
the probability of \(\sim H\) given the absence of evidence \(\sim D\)
is greater than the probability of \(\sim H\) given the absence of the absence of evidence \(\sim \sim D\) supporting \(H\), i.e., evidence \(D\) supporting \(H\). This of course does not match the maxim (1). So I do not agree that the author has proven that “absence of evidence is evidence of absence,” even though there are several other claims on the Internets to the contrary: see here, and here, and here.
The author also has the following simplified explanation, which shows some errors in his or her thinking.
- Women and men wear skirts, and things other than skirts.
- More women than men wear skirts.
From this, we claim that observing a person wearing a skirt provides evidence that the wearer is a woman.
Now, we observe someone not wearing a skirt.
Based on the preceding, we decide that that person is more likely a man than a woman.
First of all, I don’t see the “absence of evidence” here. Is it the fact that we did not observe a skirt, but pants? That is still evidence because it provides support against the hypothesis we see a woman, not a man. We would have an absence of evidence, however, if we observed instead a naked person. This shows the necessity of defining the entire sample space (in the first definition above), which makes no mention that it is possible for people to be seen naked.
Finally, the second problem with this scenario is, as said above, the lack of priors. What if there were 5 billion females, and 1 male in our sample space? What does that do to our confidence that since we have not observed a skirt, but pants, we have here a man?
My world remains unshaken, but still I wonder, maybe I missed something obvious? It is the inductivist in me.