Agreement as Evidence

When Fritz was here, we spent some time talking about the evidential value of agreement. We talked about the possibility of some perhaps formal results that yield something like the following: when sources of information are unreliable and independent and yet agree about some claim (regarding which they are unreliable), their agreement is evidence for the claim in question.

I’m skeptical about the epistemic value of agreement, but am interested to know what the precise principles are here that are defensible. One issue in the above principle is whether the sources have to be both generally unreliable and unreliable about the specific subject matter; another issue is the notion of independence being appealed to. And another issue is what happens when the information in the antecedent of such a principle is brought to the attention of people who take themselves to have really good evidence for the opposite point of view (that issue, of course, takes us away from the incremental issue involved in the above principle to the issue of what the total evidence confirms).


Agreement as Evidence — 26 Comments

  1. Maybe Condorcet’s jury theorem will interest you. I believe it goes as follows.

    Suppose there are N people, all of whom have the same reliability R with respect to some proposition P. That is, for each person, the probability she will say that P, given that P, is R. Suppose moreover that the people’s reliabilities are all independent. Then, (a) if P>0.5, then as N goes to infinity, the probability that the majority will say that P, given that P, goes to 1; and (b) if P

  2. Ooops. My comment was tuncated. (I think it was the “less than” sign that caused the problem.) Let me finish what I was saying.

    … and (b) if P is less than 0.5 then as N goes to infinity, the probability that a majority will say that P, given that P, goes to 0.

  3. That’s what I was looking for, Campbell, thanks! Is independence here just statistical independence?

    I notice that unreliability is not required here, nor is any assumption about general versus specific reliability. All that is used is conditional probability.

    On the skeptical side, though, I’m not sure how to connect the theorem with the claim that the more independent observers who agree, the better our evidence. I notice, for example, that the reliability (probability) for all has to be the same for the theorem…

  4. One other cautionary point: the theorem gives the probability that there will be agreement that P, given that P is true, whereas the original question concerned the probability that P is true, given that there is agreement that P.

  5. I notice, for example, that the reliability (probability) for all has to be the same for the theorem…

    That shouldn’t make a difference; if you have N people each of whom have at least reliability R with respect to proposition P for some R > 0.5, the probability that the majority will say that P given P should also go to 1; because that probability should be at least as great as if all their reliabilities are exactly R. (I say “should” to signal that I haven’t done the math.)

    I’m curious as to why you’re skeptical about the epistemic value of agreement, Jon. The epistemic value of testimony is currently uncontroversial, I take it; and it seems to me that agreement is like testimony except that it corroborates already held views rather than providing evidence for new views. If we get a glimpse of someone and I think, “I’m not absolutely positive, but that looked like Ortcutt,” and I say to you “Who was that?” and you say “I’m not absolutely positive, but it looked like Ortcutt,” then my confidence that it’s Ortcutt will go up.

    Of course there may be domains where testimony is more problematic, such as moral or philosophical discourse; in which arguments for authority aren’t clearly worthwhile. Then the epistemic significance of agreement is less clear to me. (Of course lots of people have done work on those fields too; don’t mean to dismiss them! It just seems to me that there’s an issue there that there isn’t about straightforward empirical questions.)

    Incidentally, this is working on preview but I vaguely remember it not working on post, but if it works: To get < type &lt;

  6. Even in simpler cases, something like that seems true. Let R1 and R2 have the same reliability n (say, .5) concerning p. Now suppose they conclude (independently and so on) that p is true: the chance that one of R1 and R2 is right and p is true is 1 – (n2) = .75. And that is in general greater than the chances that R1 (or R2) is independently right, = n = .5. So, off hand, agreement does seem to affect the evidence for p. But obviously, all sorts of other conditions would have to be met as well.

  7. That doesn’t seem right with reliability 0.5. Suppose we start out with prior probability 0.5 of p and prior probability 0.5 of ~p. Then the chance that R1 and R2 agree on p, given p, is 0.25; and the chance that R1 and R2 agree on p, given ~p, is also 0.25; so (too lazy to work out the exact Bayesian calculation) our posterior probability for p should still be 0.5. When the reliability is 0.5, you’re basically flipping coins.

    For reliability >0.5 it should work though. Say both R1 and R2 have reliability 0.6; the chance that they’ll agree on p given p is 0.36, the chance that they’ll agree on p given ~p is 0.16, so (if the priors for p and ~p were 0.5) the posterior for p should be 9/13, if I’m not mistaken. So here having two independent sources does increase your reliability over one (which would still yield a posterior of 0.6, if I’m not mistaken).

  8. Matt,

    The probablity that someone is right is .75. The probability that R1 is wrong is .5 and the probability that R2 is wrong is .5. The probability that they are both wrong is .25 (which is of course equal to the probability that they are both right). So the chances that someone is right is .75. But it does work better in the >.5 case, since the chances that they are not both right is also .75.

  9. Matt, Here’s the links to a couple of posts earlier where I wrote about my skepticism about agreement being evidence:
    Agreement Post #1
    Agreement Post #2

    The first post argues for the idea that it’s not agreement per se that is the explanation for evidence, and the second suggests that agreement, when evidentially relevant, is evidence that there is evidence for the claim in question. I’m rethinking the last issue because of the various formal results in question, but these two posts say why I’m skeptical here.

  10. Following up on Matt’s point for conditional probabilities, and addressing one earlier worry, the agreement needn’t be among those whose reliability is equal. A reliability of R1 .7 and R2 .6 relative to p, for instance, will put one’s posteriors at around .8 (I think), assuming the priors for p and ~p are .5. So Pr(p/agreement) > Pr(p/R1)> Pr(p), worries about priors aside. I’m sure there’s a theorem in the vicinity.

  11. Mike, I think I disagree with what you’ve said #9 — that the probability that someone is right .75, given the way you’ve described the situation. The problem is that R1 and R2 are saying the same thing, so the probability that R2 is right, given that R1 is right, is 1, and the probability that both R1 and R2 are right is .5 x 1 = .5. Thus, the probability that someone is right = P(R1 is right OR R2 is right) — P(R1 is right and R2 is right) = .5.

    In sum, while the witnesses may be “independent,” the correctness of one witness’s testimony is not independent of the correctness the others’ and this has to be taken into account in calculating the probability that they are all correct. I will conjecture that the probability that a number of witnesses are correct in testifying that P simply is given by the reliability of the most reliable witness among them. When we have a number of witnesses independently saying the same thing, we feel that their testimony is more likely to be correct because we think it likely that one of them is fairly reliable. When we know that some of the witnesses in the group are dependent on others for their information, this lowers our confidence in the accuracy of the group’s testimony, because we suppose that the reliability of these witnesses can be no greater on this point than the reliability of the witnesses from whom they got their information, and this narrows the pool from which the witness with the greatest reliability can be selected.

  12. Thanks Stephen. I’m not sure what you say squares with the Condorcet theorem above in (1). Maybe it does. Also running the probabilities I’ve assigned (w/ Pr(both agree/p)=.25)through Bayes’ theorem to generate some conditional probabilities gets us Pr(p /both agree) = Pr(p)= .5, as Matt noted above. No confirmation. Maybe your assignment would get the same result.
    My main worry is that the evidence that R1 and R2 have might well not be the same. So I cannot see why the additional evidence would not increase the probability of p.

  13. Mike, I believe what I’ve said is consistent with the Condorcet theorem because that theorem focuses on the probability that a group of witnesses will agree on P, given that P is true, whereas your calculations were focusing on the “converse”: the probability that P is true, given that a group of witnesses agree on P. These two conditional probabilities can’t be equated except in very limited circumstances (roughly, where there’s a guarantee that the group will either agree on P or agree on its contradictory; and that is almost never the case).

    Again, I think the fact that R1 and R2 might have different evidence simply goes to the point that whichever one has the better evidence supplies the upper bound on the probability that both witnesses are correct.

  14. I Think some of the stuff misrepresent the theorem. I can’t tell if it is just sloppiness in the talk or a misunderstanding. For example the claim above that the

    theorem focuses on the probability that a group of witnesses will agree on P, given that P is true

    isn’t actually correct. The theorem as I understand it is about the outcome of majoritarian voting procedures when every voter has a greater than even chance of being right. The theorem says that larger the number of voters the more chance that the majority vote will then also be right. (I’m going from memory about Condorcet, and maybe my memory is wrong, but at least some of the posters above also understand it this way.)

    That isn’t the same thing as saying that they will all agree on the truth of the issue at hand. In fact, if they are not all perfect at answering the question at hand, then the chance that they will all agree gets smaller as the numbers go up.

  15. Just sloppiness, MvR. My point was simply that one has to be careful to distinguish the probability that a group of witnesses will agree that P is true, given that P is true, from the probability that P is true, given that the group agrees that P is true. These aren’t in general same. Invoking or analogizing to Condorcet’s jury theorem in this context threatens to confuse these two conditional probabilities.

  16. A. Condorcet’s Jury Theorem

    This is Condorcet’s Jury Theorem:

    – Suppose you have some number of people who each judge between two alternatives, their judgements are probabilistically independent of each other, each judge has a prior probability of v (for “verite”) of being correct and a prior probability of e (for “erreur”) of being wrong. The majority of the judges agree on one alternative. Let h be the number of judges in the majority and k the number in the minority.
    – Then the probability that the majority are correct is:

    v^(h-k) / [v^(h-k) + e^(h-k)]

    (That’s v to the power of (h-k), divided by the sum of v to the (h-k) and e to the (h-k).) Condorcet notes that when we assume individual judges are just 80% reliable and the majority outnumbers the minority by as little as nine persons, the probability of the majority being correct exceeds 99.999%.

    B. Reliability

    In my “Probability & Coherence Justification” (Southern J Phil 1997; this was before I knew of Condorcet’s theorem), I argued that, given probabilistic independence, agreement enhances probability if and only if you assume some degree of individual reliability. By this I mean you have to assume that the probability of some proposition being true given that one of the witnesses asserts it is higher than the prior probability of the proposition. Olsson’s recent Against Coherence contains a more general proof of this result. (It’s actually a pretty simple result.) It also follows directly from Condorcet’s theorem (if e=v, then the probability of the majority being correct is only 1/2, which is no better than chance, no matter how large the majority).

    In my 1997 paper, I took this essentially to refute coherentism, because I thought it meant that in order to have coherence provide justification for something, you have to have at least some degree of independent (foundational) credibility for the individual beliefs.

    C. Independence

    The independence assumption mentioned in (B) is actually conditional probabilistic independence: that is, we assume that, given that X is true, one judge’s asserting X has no effect on the probability of another judge asserting X; also, given that ~X, one judge’s asserting X has no effect on the probability of another judge asserting X.

    This assumption rules out scenarios like this: Two witnesses get together and agree on what their story is going to be before talking to us. In this case, obviously the fact of agreement doesn’t do anything to enhance the credibility of their story.

    However, as I show in a forthcoming paper, the conditional independence assumption is not needed. Agreement can enhance credibility provided that the following holds:

    P(W2|W1,X) > P(W2|W1,~X)

    where X is the proposition agreed upon by the witnesses, W1 is the proposition that the first asserts X, and W2 is the proposition that the second witness asserts X. If that inequality holds, then reliability is not needed. If the witnesses individually have zero credibility (their individual testimonies have no effect on the probability of what they assert), then their agreement enhances the probability of what they assert if and only if the above inequality holds.

    That inequality essentially says that W1 is more positively (probabilistically) relevant to W2 on the assumption that X is true than it is on the assumption that X is false. So, for example, the ideal case is one in which the witnesses would definitely agree if X is true, but would either disagree or would agree on ~X if X is false.

  17. There are several versions of the Jury Theorem available. A few years back I worked out a version that only requires that the “average voter competence” is greater than 1/2 (rather than requiring that each voter’s competence is greater than 1/2). And that version also does not require (stochastic) voter independence. It depends only on the covariance among voters. Even with a relatively high covariance, if the voting population is large enough, one still gets a very high probability that the majority will vote for the true hypothesis.
    As for the inverse inference, one can also prove a “convincing majorities” theorem — a version of Bayes’ theorem, that shows that under the same circumstances, the posterior probability of the true alternative becomes very large (approaching 1 for large enough numbers of voters).
    [The paper never got into print. But I can share details, if you are interested.]

  18. Michael, notice that in the version you give under 17.A two featutes are present: 1) there are only two options, and 2) the majority necessarily agrees on one option or the other. Under these circumstances, the probability that the majority will vote for P, given that P is true, and the probability that P is true, given that the majority votes for it, are the same. But most groups of witnesses will not satisfiy these two features.

  19. In other words, there is an ambiguity hidden in the statement of the theorem in 17.A. The definition of v — “the probability of being correct” — is ambiguous between the probability that P is true, given that a judge votes for P, and the probability that the judge will vote for P, given that P is true. The quantity e has a similar ambiguity. In this specific specific context, the ambiguity is harmless. But elsewhere it is not.

  20. Jon, thanks for the links to previous threads. I think that part of the issue may come in defining “simple agreement”; should this be a case in which you know that the other people are working from the same evidence, or one in which you know nothing other that that they agree? Interesting questions.

    Here’s a case in which it seems pretty clear that agreement doesn’t provide any evidence. We’ve been watching someone drawing balls out of an urn, with replacement; 90 out of 100 draws have been red. I think, “The next draw will be red.” I say to you, “Do you think the next draw will be red?” You say, “yes.” We’re each 90% reliable, but our agreement doesn’t raise our reliability to 81/82; pretty clearly there’s a violation of independence here. (I think; here I’ll take agreement as much needed evidence!)

  21. Good points, Stephen and Jim. I was just quoting the simple version of the theorem I found in the introduction to Condorcet: Foundations of Social Choice and Political Theory (pp. 35-6)

    Jon —
    I don’t have it posted, but I’ll send you a copy. It’s for a special issue of Synthese on coherence & probability edited by Olsson. I pretty much take back my earlier ‘refutation’ of coherentism and criticize Olsson’s objection to coherentism, both by questioning the conditional independence assumption.

    Anyway, here’s an interesting applied epistemology issue related to this: the “two billion believers can’t be wrong” argument. This is occasionally used on behalf of religion: so many people believe in God, they can’t all be wrong, can they? If you apply Condorcet’s Jury Theorem, the results would be impressive.
    The problem is that individuals’ beliefs aren’t probabilistically independent (e.g., an individual’s religious opinion is a very strong predictor of his children’s). It’s also not completely obvious that individuals are more reliable than chance when it comes to religious opinions.

  22. This argument appears to leave out the implications of our social context. The obvious argument here is Hitler and the German people in the early 1930’s.

Leave a Reply

Your email address will not be published. Required fields are marked *