University students make up the majority of the samples for subject-based research articles in experimental psychology, mainly because they are cheap and convenient, but the reliance on student samples raises the question of whether results based on those samples are representative of the general population. An interesting letter on the subject appeared in Association for Psychological Science in 2006, titled “How Random is That?”
There are important exceptions: convenience sampling seems fine for studying cognitive functions, for instance. But in most non-cognitive areas of psychological research, sampling bias is an issue.
Which brings me to X-phi and the reliance on convenience sampling for assessing folk attitudes about philosophical intuitions. Why think that your samples are representative of the folk? And, more to the methodological heart of the matter, perhaps some caution might be needed before finger-wagging about appeals to intuition. For, if standard practices within (non-cognitive) psychology are what is being held up as the example to follow, the tables may very well be turned on you:
Feldman Barrett admits that some psychologists aren’t careful enough about making generalizations — in part because they lack a sufficient understanding of statistics. “Most of us are not trained in sampling theory,” she said. “There are sciences that take sampling much more seriously than we do.”
Michael Hunter is trained in sampling theory, and he does take it seriously. Co-author of the 2002 paper “Sampling and Generalizability in Developmental Research,” Hunter also sees this shift away from statistical methodology in psychological research, away from a time when using analysis of variance supported an experiment’s causality and using multiple regression analyses helped infer generalizability.
“Today, and for some time now, psychological researchers judge generalizability not on statistical grounds but instead on methodology, educated guesses or common sense, and accumulated knowledge,” Hunter said. Some of these techniques are fine to draw generalizations from, he said — sometimes even highly effective — as long as they are based on random samples of the population in question. The problem, of course, is that college students are seldom a random sample of a university population, let alone a national one. “Obviously, this basis of generalizability does not augur well for the generalizability of some research with college students, who are a selective population and who are rarely randomly sampled” (“How Random is That?“, APS 2006; emphasis, mine).