Re: Evaluating taggers

Mark Johnson (
Mon, 17 Jul 1995 10:38:05 +0000 (EDT)

It seems to me that theoretical concepts such as tag sets, bracketting
conventions or whatever, can only be evaluated with respect to a
particular task.

In old-fashioned generative linguistics, that task was separating
putatively grammatical from putatively ungrammatical sentences. A
particular category distinction (e.g., between `verb' and `auxiliary')
was supposed to be justified by the role it played in the theory as a
whole. Since there are many very different ways of constructing such
a theory, some theories will support certain category distinctions
that other theories will deny (`verb' and `auxiliary' is actually such
an example).

I suppose one could try to ask the harder question ``What category
distinctions must any adequate account make?'', but I am skeptical
that there is a theory-independent answer. For one thing, theoretical
assumptions about the interaction of morphology, syntax and semantics
would probably influence the kinds and structure of the category

The same thing seems to be true about bracketting, despite the
putative psychological claims made by some theorists. Claims that a
certain style of bracketting is `theory-neutral' seem to me to be more
sociological rather than scientific in nature: i.e., a
`theory-neutral' bracketting is one which hopefully a majority of
contemporary linguists would more or less assent to.

I think it would be very useful to evaluate the utility of various tag
set distinctions and kinds of bracketting for _quantified_ tasks.
While I suspect that ultimately one needs to evaluate particular tag
sets for particular tasks (e.g., ``modifying the tag set so that it
distinguishes between tensed and untensed verbs decreased the error
rate in our speech recognizer by XX%''), it might be worthwhile
systematically examining the effect of certain tag set distinctions on
the cross entropy of certain kinds of models. For example, on a
backed-off tri-tag model does splitting the tag `verb' into
`auxiliary' and `main verb' improve performance?



Mark Johnson, Cognitive & Linguistic Sciences Box 1978 Brown University, Providence, RI 02912 telephone (401) 863 1670, telefax (401) 863 2616. preferred email: