***************************************************************
Recognising Textual Entailment (RTE) Challenge
A Common Benchmark for Semantic Inference
Schedule Change and 2nd Call for Participation
http://www.pascal-network.org/Challenges/RTE/
***************************************************************
NOTE: The RTE Challenge schedule has been shifted by a few months
due to schedule changes in the general PASCAL Challenges Program.
Recent years have seen a surge in research of text processing applications
that
perform semantic-oriented inference about concrete text meanings and their
relationships. Even though many applications face similar underlying
semantic
problems, these problems are usually addressed in an application-oriented
manner. Consequently it is difficult to compare, under a generic evaluation
framework, semantic methods that were developed within different
applications.
The PASCAL RTE Challenge introduces textual entailment as a common task
and evaluation framework for Natural Language Processing, Information
Retrieval and Machine Learning researchers.
Textual entailment recognition is the task of deciding, given two text
fragments,
whether the meaning of one text is entailed (can be inferred) from the other
text.
Participants in the evaluation exercise will be provided with pairs of small
text
snippets, which we term Text-Hypothesis (T-H) pairs. The entire data set
includes over 1000 English T-H pairs from the news domain (political,
economical, etc.), split into development and test subsets. Examples were
manually tagged for entailment (i.e. whether T entails H or not) by human
annotators. Participating systems will have to decide for each T-H pair
whether T
indeed entails H or not, and results will be compared to the manual gold
standard.
The dataset was collected with respect to different text processing
application
settings, such as question answering, information extraction, information
retrieval, multi- document summarization, paraphrase acquisition, and
machine
translation. Each portion of the dataset includes typical T-H examples that
correspond to success and failure cases of such applications. The examples
represent different levels of entailment reasoning, such as lexical,
syntactic,
morphological and logical.
The goal of the challenge is to provide a first opportunity for presenting
and
comparing possible approaches for modeling textual entailment. In this
spirit, we
aim at an explorative rather than a competitive setting. While participant
results
will be reported there will not be an official ranking of systems. Partial
coverage
submissions, for systems that handle only specific parts of the test
examples, are
allowed. The development subset is released first to give an impression of
the
different types of test examples, but systems are likely to use
external/unsupervised knowledge resources.
Examples
-----------------------------------------------------------
TRUE
C&D Technologies announced that it has closed the acquisition
of Datel, Inc.
C&D Technologies acquired Datel Inc.
-----------------------------------------------------------
FALSE
The National Institute for Psychobiology in Israel was
established in May 1971 as the Israel Center for
Psychobiology by Prof. Joel.
Israel was established in May 1971.
-----------------------------------------------------------
FALSE
Reagan announced he had Alzheimer's disease, an incurable
brain affliction.
Reagan died of pneumonia complicated by Alzheimer's disease
-----------------------------------------------------------
TRUE
Eyeing the huge market potential, currently led by Google,
Yahoo took over search company Overture Services Inc last year.
Yahoo bought Overture.
-----------------------------------------------------------
Important Dates (may be subject to change)
----------------------
Full Development Set Available August 30, 2004
Release of Test Set Dec 15, 2004
Deadline for Participants'
Results Submissions February 15, 2005
Release of individual Results February 25, 2005
Submission of Participants'
Reports March 11, 2005
PASCAL Challenges Workshop
(in Southampton, U.K.) 11-13 April 2005
Organizing Committee
-------------------------------
Ido Dagan, Oren Glickman (Bar Ilan University, Israel)
Bernardo Magnini (ITC-irst, Istituto per la Ricerca
Scientifica e Tecnologica, Italy)
For registration, further information and inquiries - visit
the challenge web site
(http://www.pascal-network.org/Challenges/RTE/) or contact
Oren Glickman <glikmao@cs.biu.ac.il>.
_______________________________________________
Elsnet-list mailing list
Elsnet-list@mailman.elsnet.org
http://mailman.elsnet.org/mailman/listinfo/elsnet-list
This archive was generated by hypermail 2b29 : Tue Sep 07 2004 - 11:21:32 MET DST