Second Call for Papers
Workshop on Reading Comprehension Tests as Evaluation for
Computer-Based Language Understanding Systems
Thursday, May 4th, 2000, Seattle, Washington, USA
(post-conference workshop in conjunction with ANLP-NAACL2000)
Reading Comprehension tests, such as the one below, are designed to help
evaluate a reader's understanding of a text passage.
How Maple Syrup is Made
Maple syrup comes from sugar maple trees. At one time, maple
syrup was used to make sugar. This is why the tree is called a
"sugar" maple tree.
Sugar maple trees make sap. Farmers collect the sap. The best
time to collect sap is in February and March. The nights must be
cold and the days warm.
The farmer drills a few small holes in each tree. He puts a
spout in each hole. Then he hangs a bucket on the end of each
spout. The bucket has a cover to keep rain and snow out. The sap
drips into the bucket. About 10 gallons of sap come from each
hole.
1. Who collects maple sap? (Farmers)
2. What does the farmer hang from a spout? (A bucket)
3. When is sap collected? (February and March)
4. Where does the maple sap come from? (Sugar maple trees)
5. Why is the bucket covered? (to keep rain and snow out)
Such tests exist in many languages, have human performance benchmarks
associated with them, and come in a variety of types (short-answer,
multiple choice) and levels of difficulty. In addition, they are
generally written to make each story and set of questions
self-contained, in order to require as little outside knowledge as
possible to answer the questions.
The focus of the proposed workshop will be to explore the following
questions:
- Can such exams be used to evaluate computer-based language
understanding effectively and efficiently?
- Would they provide an impetus and test bed for interesting and
useful research?
- Are they too hard for current technology?
- Or are they too easy, such that simple hacks can score high,
although there is clearly no understanding involved?
The most direct method of exploring these questions is to choose a set
of tests and build a system that takes these tests. Some preliminary
results indicate that such tests are tractable, but not trivial and
that linguistic processing is helpful (Hirschman, et al. ACL-99). A
test set, evaluation routines, prototype system, and documentation are
available upon request to light@mitre.org.
We hope that a number of submissions will present results based on
actual reading comprehension systems. In addition, we encourage
submissions that report on other kinds of tests or similar tests in
other languages, or that address our list of questions by other
means. Note that submissions are encouraged that describe work in
progress with preliminary empirical results.
Invited speaker:
Karen Kukich (Educational Testing Service)
"NLP Tools for Identifying Reading Comprehension Skills"
Format for Submission
Authors are asked to submit previously unpublished papers only; a
workshop proceedings will be published. Our target submission length
is 2000 words but both shorter and longer submissions will also be
considered. Electronic submission of postscript will be accepted.
Hard copy submissions should include 4 copies of the paper. Since the
papers will be reviewed anonymously, please do not place the author
name on the paper. Instead include a separate title page with title,
abstract, author, and e-mail address. Unless requested otherwise,
notification of acceptance will be sent electronically to the first
author. Parallel submission is unproblematic; however if your paper
is accepted to this workshop and you decide to present it here, we
will ask you to withdraw it from any other events.
Important Dates
Deadline for submission: February 11th, 2000
Notification of authors: March 1st, 2000
Final versions due: March 10th, 2000
Address for Submission and Further Information
Marc Light
The MITRE Corporation
202 Burlington Rd.
M/S K329
Bedford, MA 01730
USA
Phone: 1-781-271-5579
light@mitre.org
(The mailing list, read-comp@linus.mitre.org, has been set up to
discuss reading comprehension tests as evaluation for computer-based
language understanding systems. It is open subscription and
unmoderated. To subscribe, send email to majordomo@linus.mitre.org
with 'subscribe read-comp' in the body.)
Program Committee:
Eric Brill
Eugene Charniak
Mary Harper
Marc Light (chair)
Ellen Riloff
Ellen Voorhees
This archive was generated by hypermail 2b29 : Tue Feb 08 2000 - 11:47:43 MET