What Experimental Philosophers Actually Do: A Quantitative Study

These past few years have seen a surge of metaphilosophical work exploring the implications (or lack thereof) of the empirical studies being conducted in experimental philosophy. Much of this metaphilosophical work is being done by people who are not experimental philosophers themselves but who simply want to address the larger philosophical questions that certain kinds of empirical research can raise. This development is very welcome indeed, and I have gotten a lot out of reading many of these contributions.

However, I do have some misgivings about one aspect of this work. Typically, papers in this tradition start out by (a) briefly describing the sort of empirical research that experimental philosophers do and then proceed to (b) engage in a discussion of the larger philosophical implications of such research. In my experience, the discussion of philosophical implications usually shows a truly impressive degree of clarity and rigor, but the description of the actual research can sometimes be a bit inaccurate. (Thus, there is a real danger that one will end up with a impressively rigorous analysis of the implications of something that experimental philosophers don’t actually do.)

To address this issue, I thought it might be helpful to conduct a quantitative study that explored the question as to what experimental philosophers have actually been doing.

To complete this study, I teamed up with our unstoppable psychology student Ike Silver. First, Silver looked through all the papers listed in the PhilPapers database under the category ‘experimental philosophy’ over the period that was then the past five years (2009-2013 and forthcoming). In total, there were 379 papers. From these papers, he extracted all of the empirical studies. This method yielded a dataset of 453 studies.  I went through these studies and coded them to get a sense for what sort of thing they purport to accomplish. (Obviously, we do not have all of the studies conducted during this time period, but we do have such a large number that the frequencies within our sample are unlikely to differ too substantially from the frequencies within the population as a whole.)

A number of experimental philosophers present their work as contributing to a ‘negative’ program, which aims to show that people’s intuitions are fundamentally unreliable. These experimental philosophers have always described their approach, extremely accurately, as just one among the various different projects within the broad field of experimental philosophy. However, some metaphilosophical work by non-experimental philosophers seems to proceed on the assumption that this negative program is the main focus of contemporary experimental work. This assumption turns out to be wildly incorrect. In actual fact, contributions to this negative program account for only 1.1% of empirical studies in experimental philosophy in the past five years.

The remaining 98.9% of studies aim to make some sort of positive contribution. The studies break down by topic as follows:


[Note: The ‘anti-negative’ category consists of experiments presented as providing evidence against the negative program (e.g., replication failures).]

Zooming in now on the studies on epistemology specifically, the breakdown looks like this:

38.8% Contextualism/IRI

22.5% Epistemic Side-Effect Effect

20% Does Knowledge Entail Belief?

7.5% Gettier Cases

11.3% Other

Of course, the relevance of these facts to questions in metaphilosophy is limited. Philosophers are free to ask any questions they want to about the philosophical implications of various forms of empirical research. Metaphilosophical work of this type might lead to important insights, and those insights would not be in any way called into question by the sorts of facts I have been presenting here.

Still, it does seem that these facts are relevant to the degree that people working in metaphilosophy are making general claims about experimental philosophy. If one wants to make claims about experimental philosophy — especially if one wants to claim that experimental philosophy is fundamentally bankrupt or otherwise mistaken — then, surely, one has to engage with the sort of research that experimental philosophers have actually been doing!

[I present these results in a forthcoming paper, where I try to argue for a metaphilosophical framework aimed at capturing what experimental philosophers typically do.]


What Experimental Philosophers Actually Do: A Quantitative Study — 8 Comments

  1. Joshua, this is very interesting, indeed: thank you very much for sharing it!

    Can I ask you if you know of any studies about the reverse question?

    Here is what I have in mind. I tend to think of myself as a philosopher with an interest in metaphilosophical, or methodological, questions and a genuine admiration for work in experimental philosophy (although I am familiar with only very little of this work, and understand even less of it).

    However, I sometimes wonder to what extent work in experimental philosophy, or maybe I should say: experimental work on intuitions, is relevant to what philosophers actually do, or to what philosophers actually think they are doing when they philosophize.

    Several philosophers (Cappelen, Philosophy Without Intuitions, OUP 2012; Hintikka, The Emperor’s New Intuitions, JPhil 1999) are fairly critical of the use of intuitions in philosophy, and the only role (I am aware of) intuitions play for me these days is that I try to think of ways how we can avoid relying on them.

    I’d be very interested to learn if experimental philosophy’s assumptions about the importance of intuitions for philosophical methodology match with what philosophers actually think about the role of intuitions for what they are doing. (Whatever they think, they may, of course, be wrong, but that’s another question — as is the normative question how philosophy should be done.)

  2. Thanks Franz! This is a really great question, which gets right to the heart of the issue.

    As you note, Cappelen and others have argued that philosophers don’t actually appeal to intuitions and that this undermines work in experimental philosophy that seeks to criticize such appeals. This argument has set off a very interesting debate, and helpful points have been made on both sides.

    The thing I want to call attention to, though, is that Cappelen’s argument only applies to a very small part of experimental philosophy, namely, the part that criticizes philosophers’ (alleged) appeals to intuition. For the whole rest of experimental philosophy, it doesn’t really matter whether philosophers appeal to intuitions or not.

    As one example, consider Jonathan Schaffer’s experimental work in epistemology. Schaffer is interested in the semantics of knowledge ascriptions, and he therefore gathers empirical data about how people use the word ‘know.’ In evaluating this research, it clearly doesn’t matter at all whether previous philosophers have appealed to facts about how people use the word ‘know’ or even whether they have taken an interest in the semantics of knowledge ascriptions. The only thing that matters is whether the empirical facts that Schaffer uncovers actually *do* provide evidence for the semantic claims he defends. (Of course, you might say that they don’t, but you presumably wouldn’t argue for that view on the basis of facts about what philosophers typically do or are interested in.)

    Then the point I was trying to make is that the overwhelming majority of experimental philosophy is more like this last example. For example, Jonathan Schaffer has 26 experiments in the five-year dataset, which is more, all by itself, than the entire research program Cappelen is trying to criticize.

  3. This is extremely helpful, thank you very much, Joshua!

    I am not familiar with this work by Jonathan Schaffer, but I am certain that it is as brilliant as his work that I am familiar with.

    However, I continue to be inclined to think that this work in experimental philosophy has little to do with philosophy (in this case: epistemology). To study how people talk about knowledge is a very interesting question, as are many other linguistic questions. And experimental data on linguistic intuitions are highly relevant to these linguistic questions.

    However, these data don’t become relevant to philosophy by replacing a philosophical question about the nature of knowledge with a linguistic question about how people talk about knowledge. To give you an example, a philosophical question is that of skepticism. Data showing that people talk as if skepticism were false do not show that skepticism is false. More generally: data to the effect that people talk as is p do not show that p.

    Or maybe a different example illustrates the point better. Religious talk is very interesting and can be studied empirically, but it has no relevance for the truth-value of religious questions: you don’t (dis)prove God’s existence by gathering data about how people talk about deities. For that you need an argument! Similarly, causal talk is very interesting, but I fail to see what it tells me about the nature of causality (if there is such a thing in the first place). And ditto for knowledge: the way people talk about knowledge tells me little about the real thing.

    To be sure, I do not want to say that these linguistic questions are not interesting, or that you or Schaffer or anyone else would not carefully distinguish between talk about p and p.

    However, for you to point to experimental data about how people talk about knowledge, and then say that this illustrates how experimental philosophy can be relevant to epistemology, is, in the present context, not ideal: as I said, I am certain that Schaffer’s work on the semantics of knowledge ascriptions is as brilliant as his work that I am familiar with, but it is not work on knowledge.

  4. Hi Franz,

    Thanks so much for this very thoughtful and measured response. I really appreciate the way you are engaging with this work in a critical but also very constructive spirit.

    The main thing I want to say about the questions you raise is that they are *exactly* the right questions to be asking. You are getting right at the core of the sort of thing experimental philosophers have actually been doing and are asking precisely the key metaphilosophical question that they will have to answer in order to justify their work.

    Broadly speaking, it seems that defenders of experimental philosophy have tried to justify this type of research in one of two ways.

    The first is to say that empirical research actually can help in addressing the sorts of questions you bring up here. For example, many philosophers have suggested that we can get some defeasible but still helpful evidence about the nature of free will by studying people’s ordinary way of thinking about free will. (The usual thought there is that if we can get a better understanding of the cognitive processes that draw people toward one answer and those that draw them toward the other, we will be better able to say which of these processes is more worthy of our trust.) I realize that you are unlikely to find this answer compelling, especially in so compressed a form, but I thought it might be helpful at least to gesture at the outlines of an answer to your worry.

    Then the second is to say that the sorts of issues addressed by this empirical research are philosophically important in and of themselves. To give one example: Seth Yalcin, Zoltan Szabo and Justin Khoo have been doing work on empirical questions about modal language. The idea there is not that we can use empirical facts about language to get indirectly at some metaphysical question about modality; it is that the nature of human language is itself philosophically important. Or to take an especially prominent example: Shaun Nichols has been pursuing a huge research program devoted to understanding moral psychology. The idea behind that work is not just that an understanding of moral psychology can help us to figure out which things truly are morally right; it is that moral psychology is philosophically important in and of itself.

    Of course, in saying all this, I don’t at all mean to suggest that there are no good objections to these two research programs, and I would be very interested to hear further criticism of the sort you have been developing.

  5. Thank you so much, Joshua! These are very helpful responses!

    I agree with every single aspect of the second response, although I would perhaps stress a bit more that work in moral psychology is work in, well, psychology (rather than moral philosophy), and that studying modal language is studying, well, language (and not modality).

    To be sure, I don’t think we should stress these differences because one set of questions would be more interesting or important or worthy of funding than another, but because doing so will help in the study of either.

    I have to admit that I find it a bit more difficult to see how the first response can work, although you have formulated in a way that is hard to resist. The burden of proof is on me here, though (meaning, I have to study these studies before I can make any negative claims), and this will be something that one has to check from case to case. Maybe this is possible in the case of free will, which is a topic I am not familiar with.

    For the topics I am somewhat familiar with, such as counterfactuals and causation and knowledge and skepticism, I have not found the empirical studies about how we think or talk about these things to provide any evidence about the things themselves. Again, this is not a problematic thing, and does not in any way detract from the importance of this work. It’s just something I think we should keep in mind when we study either these things themselves, or the way we talk and think about these things.

    Most importantly, thanks so much for time and patience and what has effectively been a one-on-one tutorial in x-phi!

  6. I feel like in some sense things have moved on and there is not much point in fighting revisionist history. What’s important is that people have moved on from the negative program, so much so that people deny being associated with it. Good. But for those of us who were caught in it, the point of targeting certain programs, such as contextualism and interest-relative invariantism, was precisely because those programs did, or seemed to, ascribe great weight to intuitions. The people writing those papers were not immersed in epistemology; they wanted to make points about the method of intuitions. I was commenting and engaging with all of them, and they were criticizing my work. Things have moved on from there. Turri, Buckwalter, Sripada, Schaffer, and many others have positive programs. That’s great.

  7. I think that a lot of advocates of intuition-reliance saw the first couple of papers in X-phi as directed against their cherished methodology (which they explicitly were, im thinking of the famous 2001 paper specifically). Then there was the impression that, above and beyoned cultural and socioeconomical sensitivities exhibited by non-philosopher respondents,the job of X-phiers were now to find more of these irrelevant sensitivies (ordering, gender) to challange traditional methodology. To many, that defined the whole project. Today I think there is a different attitude, with many intuition proponents just defaulting the expertise-argument.

  8. Hi Martin,

    I completely agree with the main point you make here. Many non-experimental philosophers probably looked at a few early papers in experimental philosophy and came away with a very definite impression of what experimental philosophy was all about. They may then have retained that first impression even when the vast majority of subsequent papers turned out to be doing something different.

    In thinking about this difference, I guess I don’t think it would be helpful to suppose that the authors of the later papers are in any way rejecting the claims made in the earlier papers. The point is just that they are taking up entirely different questions and addressing them in very different ways.

    Actually, the previous commenter, Jason Stanley, is a good example. Jason has a very nice experimental philosophy paper:


    If you take a look at that paper, you’ll see that it isn’t an attempt to argue for any particular claim regarding the question as to whether people’s intuitions are affected by irrelevant factors. It’s just about an entirely separate question, namely, whether knowledge attributions are affected by stakes. (The vast majority experimental philosophy papers have that same quality.)

Leave a Reply

Your email address will not be published. Required fields are marked *