Sponsored by ACL SIGLEX and EURALEX
There are now many automatic Word Sense Disambiguation (WSD) programs
but it is currently very hard to determine which are better, which
worse, and where the strengths and weaknesses of each lie. There is
widespread agreement that the field urgently needs an evaluation
framework. Under the auspices of ACL SIGLEX and EURALEX, a pilot will
take place in the course of 1998. As in ARPA evaluation exercises,
the framework comprises:
1) definition of task and scoring metric
2) preparation of a set of manually tagged correct answers
3) a dry run, with sample data distributed to participants
4) distribution of test data to participants;
participants sense-tag and return;
taggings scored against correct answers
5) workshop to discuss results, lessons learned, way forward
We shall be undertaking evaluation for at least English, French,
Italian and Spanish. The workshop will be held at
Herstmonceux Castle, Sussex, UK
on
Sept 2-4 1998
If you have a working WSD program (or will have one by Summer 1998),
and would like to subject it to objective, quantitative evaluation, or
if you have skills or resources that you would like to contribute to
the exercise, first look at
http://www.itri.bton.ac.uk/events/senseval/exp-interest.html
and then mail your expression of interest to
senseval-coord@itri.bton.ac.uk
Timetable
=========
As soon as possible:
expressions of interest to be registered
By end of:
April 98: 'dry run' data samples distributed to participants
June 98: test data distributed to participants, and, one week
later, returned with taggings
July 98: correct answers and scores made available
Sept 98: workshop
The workshop will comprise:
(1) reports/papers on the manual tagging and other aspects of
the evaluation exercise
(2) reports/papers from participants on the design and
performance of their system
(3) other related research papers
(4) working sessions on the way ahead for WSD evaluation
The deadline for "other related research papers" is
16th April 1998
* Preference will be given to papers discussing WSD, with particular
attention to evaluation issues
* Maximum submission length: 6 pages
* First page to include title, abstract, and author's name(s) and
contact details
* Electronic submission of postscript documents permitted
but must be supported by hard copy to arrive not later than 23rd April
(in case of printing problems).
email: senseval-submissions@itri.bton.ac.uk
hard copy: SENSEVAL Submissions
ITRI
University of Brighton
Lewes Road
Brighton BN2 4GJ, UK
Full details available at http://www.itri.bton.ac.uk/events/senseval/
Adam Kilgarriff
SENSEVAL co-ordinator