This seems like a sensible idea. Not to let one shameless plug go by
without a second in kind, :-) this is an issue that comes up even more
forcefully in evaluating word sense tagging, in comparison to which
the POS-tagging community seems virtually standardized. David Yarowsky
and I made a proposal addressing this and other evaluation issues that
was adopted, with modification and elaboration, in last fall's
SENSEVAL exercise, organized by Martha Palmer, Adam Kilgarriff, and
Joseph Rosenzweig (soon to be covered in a special issue of _Computers
and the Humanities_). Some of the points we made referred
specifically to word sense disambiguation, but others, in particular a
perplexity-like evaluation metric, would seem to apply equally well
here. The original paper (in the 1997 ANLP SIGLEX workshop) is
<A HREF="http://umiacs.umd.edu/~resnik/papers/siglex97_perspective.ps">
Philip Resnik and David Yarowsky, A perspective on word sense
disambiguation methods and their evaluation"</A>, position paper
presented at the <A
HREF="http://www.sfs.nphil.uni-tuebingen.de/~light/semtag_ws.html">ACL
SIGLEX Workshop on Tagging Text with Lexical Semantics: Why, What, and
How?</A>, held April 4-5, 1997 in Washington, D.C., USA in conjunction
with ANLP-97. <P>
and the SENSEVAL page is http://www.itri.bton.ac.uk/events/senseval/.
Philip
----------------------------------------------------------------
Philip Resnik, Assistant Professor
Department of Linguistics and Institute for Advanced Computer Studies
1401 Marie Mount Hall UMIACS phone: (301) 405-6760
University of Maryland Linguistics phone: (301) 405-8903
College Park, MD 20742 USA Fax : (301) 405-7104
http://umiacs.umd.edu/~resnik E-mail: resnik@umiacs.umd.edu