Volume 7
The Evaluation of Multiple Post-Editors:
Inter-Rater Consistency in Correcting Automatically Tagged Data
John Paul Baker
Abstract
The experiment investigated the hypothesis that using human
post-editors to check automatically tagged corpora would introduce
inconsistencies in the data. Nine experienced post-editors were
given sentences of written and spoken data, which had previously
been tagged by CLAWS and were asked to remove errors from the output.
Once ambivalent words had been removed from the data, mean rater
accuracy was found to be higher than the accuracy of CLAWS output
(98.7% to 95.3%), while overall consistency between post-editors
was 98%. As a result of the experiment, ambivalent cases were
resolved through the incorporation of new guidelines. It was also
found that if subjects made a slip, it would be highly likely to
involve substituting or leaving a noun tag in the place of the
correct tag.
This paper is available at the price of 2.50 UK Pounds. Cheques in
UK currency should be made out to "Lancaster University" and sent
to the following address:
The UCREL Secretary (Technical Papers)
UCREL
Department of Linguistics and Modern English Language
Bowland College
Lancaster University
LANCASTER LA1 4YT
United Kingdom
Earlier volumes are also still available. For a complete list please
check out our World-Wide-Web site:
http://www.comp.lancs.ac.uk/computing/research/ucrel/tech_papers.html
Andrew Wilson
Joint General Editor, UCREL Technical Papers