A few years ago, I published a paper in which I not only cited, but also discussed at considerable length, an epistemology paper by Robert Hambourger. I wrote that Hambourger’s paper was “fascinating, but largely overlooked.” Several months ago, I was working on revisions to that paper of mine (for inclusion in a book I’m working on). By then I had been alerted, by this post by Jason Stanley over at Leiter’s blog, of the possibility of “Using Google Scholar to Assess the Impact of Philosophical Work.” So I thought I’d use Google Scholar to check just how overlooked Hambourger’s paper (still) is. “I wonder who, other than me, has cited Hambourger’s paper?” I thought. The results? Well, as you can see, Google Scholar reports than 9 works have cited Hambourger’s paper, which isn’t bad. (And, hey, that’s just now gone up to 10.)
But it doesn’t count me as having cited it. Why not? Does anybody know? Some potentially relevant info:
I really do cite Hambourger’s paper, and, as I said, even discuss it at considerable length. I do this in a published paper in The Philosophical Review. Mine is a 2002 publication, which should be about perfect for having been picked up by Google Scholar: not so recent that it hasn’t had time to pick it up, not so long ago that it’s some of the ancient history that GS hasn’t gotten to. The Phil Review publishes papers on-line (for those with subscriptions) as well as on paper. Google Scholar has registered the existence of my paper — and in fact lists it as having a good number of citations (50) itself — but apparently doesn’t count the citations made in my paper. And a pre-publication draft of my paper has been on-line for years at my Yale web site. I also cited Hambourger’s paper in a paper I published in 1992 in Philosophy and Phenomenological Research. That doesn’t show up, either. So, what’s going on? I really don’t know.
That’s of course just one example. But it’s indicative of what my experience has been on similar occasions. When I know that paper x cited paper y and I check Google Scholar, I have learned to no longer ask which papers other than x will be included on the list of papers citing y, but rather to ask whether x is one of the citations of y that GS happened to catch — and to be more than a bit dubious as to whether x will appear on the list.
Meanwhile, GS does often pick up on citations made in unpublished papers that are posted on individuals’ personal web sites. But not from everybody’s personal web sites. It gets many; it misses many.
This is all fine if you’re using GS for the purpose as I usually use it: as one tool among others for locating papers on topics I’m interested in. For that purpose, it’s good that it picks up works in progress on people’s web sites.
But do we really want to use something that counts citations from unpublished papers on random folks’ web sites, but doesn’t count MANY citations from published papers in reputable professional journals, as a source for assessing the impact of the writings of philosophers — and for ranking departments, as in this interesting experiment by Jon Kvanvig here on Certain Doubts? (Jon presents this with a heavy dose of warnings.)
It’s possible that GS misses genuine citations, and also gets false positives, in such a manner that “it all comes out in the wash,” and the number of GS counted citations will generally be a pretty good indication of how many citations a paper would have if citations were counted correctly (however exactly that would be). But that doesn’t seem to be how it works. Jason, for instance, has noticed, as I have, that some areas of philosophy seem to generate higher GS citations than other areas. So, Jason’s first warning: “One should refrain from comparing hit numbers across areas of philosophy.” And he lists other potential pitfalls.
But are we watching out for the right pitfalls? That depends on how GS works, and the right explanation for why it misses what it misses and gets what it gets. Until I understand that, I, for one, really don’t know what exactly to watch out for. The theory seems to be that people who work in areas of philosophy that overlap with or are of interest to disciplines other than philosophy will tend to get more citations. But I don’t have much confidence that that’s most of the explanation of what’s going on here. What I worry about is that GS is extremely patchy in what it picks up on in all sorts of ways, so what’s important to GS counts is to be cited by “well-wired” segments of academia — or segments that are “wired” in ways that GS happens to pick up on. So, several people, including Jason, have noticed that those who work in the history of philosophy tend to have low GS-counts. Is that because their work is of less interest (or at least less citation-generating interest) to non-philosophers than is the work of philosophers of language, for instance? Or is it because it is of interest to other disciplines (like, perhaps history) that tend not to be very well “wired”? Might GS be more likely to pick up on citations from papers posted on the web sites of some universities than on other universities? Given that GS counts self-citations, that would surely be important to know in assessing the relevance of higher GS scores.
So we need to know more about how GS works before we know when & to what extent to trust it as a source for assessing the impact of philosophical work. Maybe some people out there have a pretty good idea of this. (For example: GS gets citations from this, this & this journal, but not that, that or that one; it picks up on almost every citation — including self-citations — on personal web sites for these universities, but on almost none of them from those universities, etc.) But I have almost no idea of why it gets what it does & misses what it does. So I, for one, don’t know when & to what extent to trust it as a reasonable guide to the impact of scholarly work.