Article From http://blog.codalism.com/?p=817
It is common these days to see the traditional method of evaluating an information retrieval system against a test collection referred to as the “Cranfield paradigm”. For instance, Emine Yilmaz and Javed Aslam in their 2006 CIKM paper, Estimating Average Precision with Incomplete and Imperfect Judgments, denote “the test collection methodology adopted by TREC” as “the Cranfield paradigm”, and similar uses can be found in recent papers by Sakai, Scholer et al., Harman and Hiemstra, and many others besides. It is such a distinctive usage, that I came to wonder when it was introduced.
The phrase “Cranfield paradigm” does not, of course, appear in any of the Cranfield reports themselves, nor in the early literature describing the experiments at Cranfield. Contributors to Sparck Jones’s 1981 book Information Retrieval Experiment speak of work done in the same “tradition” as Cranfield (Sparck Jones, page 2), of “the ‘normal’ or archetypal retrieval test” of which Cranfield is an (but not the only) example (Robertson, page 19), or a “body of practice” based on Cranfield and later investigations (Tague, p 59), but nowhere are paradigms mentioned, nor is Cranfield even treated in a particularly paradigmatic way (despite a chapter being devoted to the Cranfield tests, and the book being dedicated to Cleverdon, the director of those tests). By the time of the 1992 Information Processing and Management special issue on information retrieval evaluation, the word “paradigm” had entered the lexicon, with Donna Harman observing in the introduction that “the test collection paradigm has … caused some major problems”, Tague-Sutcliffe declaring that “a paradigmatic shift has occurred in the research front, to user-centered from system-centered models” (page 467), and Michael Keen noting that “there is no perfect paradigm for the laboratory test” (page 491). Robertson and Hancock-Beaulieu even talk about the lack of “any kind of paradigm or consensus” regarding the concept of relevance (page 458). However, while a reference to Cranfield often lurks nearby, none of these authors actually use the phrase “Cranfield paradigm” directly. It seems that it had not yet entered mainstream usage.
The first usage of the phrase “Cranfield paradigm” appears (judging in part from Google Scholar) to be in an early, little-cited paper by B. C. Brookes, presented at SIGIR in 1980. Brookes sets out to “question the continued usefulness of what I call the `Cranfield paradigm’”, a formulation that suggests that Brookes is introducing what he believes to be a novel usage. Brookes’ paper is a discursively theoretical one, reflecting on the theory of science, Shannon’s definition of information, whether “information retrieval” should actually be called “document retrieval”, whether it should be measured on a linear or a logarithmic scale, as well as philosophical monism and dualism, the nineteenth century debate between the vitalist and physicalist schools of organic chemistry, and other such matters. He cites Bishop Berkeley, Socrates, and Einstein, describes Karl Popper’s World 3, and quotes Thomas Kuhn at length (of whom more later). Brookes ends in a manner not frequently repeated in later SIGIR papers by stating that “we need a firmer metaphysic for our studies”.
Brookes’s paper did not cause a revolution in the science of information retrieval, nor does it seem to have popularised the phrase “Cranfield paradigm” (which he repeats in a 1983 paper in the Journal of Information Science). The next usage appears to be in Towards an information logic, a paper presented by Keith van Rijsbergen at SIGIR 1989. This has a section entitled “The Cranfield Paradigm”, which van Rijsbergen defines as one in which relevance is treated as a hidden variable, only indirectly accessible by collecting data from the user, a method which “represents an extreme descriptivist approach to the science of IR” (page 78). Van Risjbergen also goes on to claim (rather boldly, given subsequent history) that the use of IR techniques in multimedia-rich environments means that “one might say that we have come to the end of the empirical era in IR” (page 79).
Van Risjbergen does not cite Brookes, so whether his use of the phrase derives from Brookes, or is an independent coinage, or is derived from somewhere else, is unclear. An author (one of the few) who does cite Brookes’ work is David Ellis, in his 1984 article Theory and explanation in information retrieval research, but without reference to the “Cranfield paradigm”. However, paradigms later become a recurrent theme in Ellis’s work, beginning in 1992 with The Physical and Cognitive Paradigms in Information Retrieval Research. Ellis traces the concept of a paradigm back to Kuhn, and then attempts to describe what makes Cranfield a paradigm in the Kuhnian sense. For Ellis, Cranfield is a “physical paradigm”: a model of the information retrieval system as a physical machine, and of the retrieval experimentation as a physical experiment. Ellis quotes Cleverdon’s description of the Cranfield approach as being like testing in a wind-tunnel to underline this point.
No comments:
Post a Comment