These past few years have seen a surge of metaphilosophical work exploring the implications (or lack thereof) of the empirical studies being conducted in experimental philosophy. Much of this metaphilosophical work is being done by people who are not experimental philosophers themselves but who simply want to address the larger philosophical questions that certain kinds of empirical research can raise. This development is very welcome indeed, and I have gotten a lot out of reading many of these contributions.
However, I do have some misgivings about one aspect of this work. Typically, papers in this tradition start out by (a) briefly describing the sort of empirical research that experimental philosophers do and then proceed to (b) engage in a discussion of the larger philosophical implications of such research. In my experience, the discussion of philosophical implications usually shows a truly impressive degree of clarity and rigor, but the description of the actual research can sometimes be a bit inaccurate. (Thus, there is a real danger that one will end up with a impressively rigorous analysis of the implications of something that experimental philosophers don’t actually do.)
To complete this study, I teamed up with our unstoppable psychology student Ike Silver. First, Silver looked through all the papers listed in the PhilPapers database under the category ‘experimental philosophy’ over the period that was then the past five years (2009-2013 and forthcoming). In total, there were 379 papers. From these papers, he extracted all of the empirical studies. This method yielded a dataset of 453 studies. I went through these studies and coded them to get a sense for what sort of thing they purport to accomplish. (Obviously, we do not have all of the studies conducted during this time period, but we do have such a large number that the frequencies within our sample are unlikely to differ too substantially from the frequencies within the population as a whole.)
A number of experimental philosophers present their work as contributing to a ‘negative’ program, which aims to show that people’s intuitions are fundamentally unreliable. These experimental philosophers have always described their approach, extremely accurately, as just one among the various different projects within the broad field of experimental philosophy. However, some metaphilosophical work by non-experimental philosophers seems to proceed on the assumption that this negative program is the main focus of contemporary experimental work. This assumption turns out to be wildly incorrect. In actual fact, contributions to this negative program account for only 1.1% of empirical studies in experimental philosophy in the past five years.
The remaining 98.9% of studies aim to make some sort of positive contribution. The studies break down by topic as follows:
[Note: The ‘anti-negative’ category consists of experiments presented as providing evidence against the negative program (e.g., replication failures).]
Zooming in now on the studies on epistemology specifically, the breakdown looks like this:
22.5% Epistemic Side-Effect Effect
20% Does Knowledge Entail Belief?
7.5% Gettier Cases
Of course, the relevance of these facts to questions in metaphilosophy is limited. Philosophers are free to ask any questions they want to about the philosophical implications of various forms of empirical research. Metaphilosophical work of this type might lead to important insights, and those insights would not be in any way called into question by the sorts of facts I have been presenting here.
Still, it does seem that these facts are relevant to the degree that people working in metaphilosophy are making general claims about experimental philosophy. If one wants to make claims about experimental philosophy — especially if one wants to claim that experimental philosophy is fundamentally bankrupt or otherwise mistaken — then, surely, one has to engage with the sort of research that experimental philosophers have actually been doing!
[I present these results in a forthcoming paper, where I try to argue for a metaphilosophical framework aimed at capturing what experimental philosophers typically do.]