Recently a number of articles, books, and reports
dealing with information systems, i.e., document retrieval
systems, have advanced the doctrine that such systems
are to be evaluated in terms of the degree or percentage
of relevancy they provide.
Although there seems to be little agreement on what
relevance means, and some doubt that it is quantifiable,
there is, nevertheless, a growing agreement that a fixed
and formal relationship exists between the relevance and
the recall performance of any system. Thus, we will find
in the literature both a frankly subjective notion of
relevance as reported by individual users, and equations,
curves, and mathematical formulations which presumably provide
numerical measures of the recall and relevance
characteristics of information systems. This phenomenon
of shifting back and forth from an admittedly subjective
and non-mathematical term to equations in which the
same term is given a mathematical value or a mathematical
definition has its ancient parallel in discussions
of probability. One cannot, of course, legislate the
meaning of a term. It all depends, as Alice pointed out,
on "who is master," the user or the term. On the other hand,
the use of a single term in the same document to cover
two or more distinct meanings, especially when such a
usage is designed to secure the acceptance of a doctrine
by attributing to it mathematical validity which it does
not have, represents a more serious situation than merely
careless ambiguity.