Keith asks about the reason behind some of the experimental design in one of the early experimental philosophy papers, and links to a draft of a paper that attempts to offer some critiques of some of those experiments. Regarding the latter, all I’ll say here is that I hope that if people read the paper, that they also read the painfully drawn-out exchange concerning it that Keith had linked to here. In short, although the author of that paper found some interesting materials out of which a possible confound might somehow be constructed, he has not yet come particularly close to actually constructing one.
I would like to answer Keith’s question, though, as to why we used the binary forced-choice probe “REALLY KNOWS ONLY BELIEVES” in the original study. The answer is that we wanted to try to make it more likely that our subjects would answer the question using a suitably epistemic construal of “knows”, of the sort that epistemologists are interested in, and not other standard usages in English like believes-with-subjective-certainty or “exam knowledge” (pace Hetherington). I think it probably did that; for example, we did include one comprehension check question, about whether someone believing correctly, but at chance, on the basis of a “special feeling”, and the subjects were nearly univocal in circling “ONLY BELIEVES”. Now, one would surely have expected — and Cullen’s data would confirm that expectation — that using those adverbs would affect the baseline rates at which subjects would attribute or deny knowledge, generally decreasing that baseline rate of attribution. But this just isn’t really a problem for our argument, since the main importance of our data is in the comparative differences, not in the absolute numbers. So, if the presence of those adverbs just suppresses attribution across the board, then it’s not really relevant to any such differences that we may have found. Moreover, we found some sorts of effects on some sorts of cases, and different ones on different cases, and none at all on some others, so it is unlikely that any broad ethnicity-based effects could explain the overall pattern of our findings. It’s going to have to come down to something about the cases themselves.
Which is not to say that there might not be something about the cases themselves that’s problematic! None of this is a reason not to not ask the kinds of questions that Keith and Cullen have asked — looking for uncontrolled confounds is one of the most important forms of interrogating someone’s report of experimental findings — but it’s important to keep an eye on what’s a substantial problem for the load-bearing parts of the argument, and what amount to more like unproblematic suggestions for refinements for possible future experiments in the area.
(Also, for what it’s worth, my more recent work with Stacey Swain and Josh Alexander doesn’t use that binary forced choice, but rather a 5-point Likert scale asking subjects the extent to which they agree with a statement of the form, “Does X know that p?” We have had a greater incidence of subjects needing to be screened out by their answer to the coinflip case as a result, which is what one would expect. I hasten to add that other recent x-phi work in epistemology, like that by Wright, Bengson, & Moffett at Wyoming on knowing how, or Phelan & Neta at UNC and Feltz & Zarpentine at Florida State on intellectualism, do not use the “really knows/only believes” formulation. So, even if it turns out that the use of the adverbs is more methodologically problematic than I have suggested here, it’s probably best to think of this as a problem that is local to my initial work with Steve Stich and Shaun Nichols, and not as any general problem with x-phi as applied to epistemology.)