Re: Corpora: Ergo's Parsing Contest

Anne Sing (annes@htdc.org)
Mon, 22 Feb 1999 14:25:43 -1000

At 01:41 PM 2/22/99 -0800, Ted E. Dunning wrote:
>The early MUC competitions (MUC-2, which dealt with telegraphic
>situation reports, in particular) have dealt with language which is
>far from the newspaper-like text which later MUC's have analyzed.
>The primary motives for this shift have been two-fold:
>
>a) first and foremost, the people with the money thought that the
>automatec analysis of newswire and similar text was of great
>importance.
>
>b) also quite important, newswire is very available and is
>unclassified. Both of these properties are quite important.

This of course is undisputed. But what about the areas of navigation
and control and q&a dialoging in educational products and games
and multi-media areas that are currently frozen at a very low level
precisely because of the lack of better NLP. If what is being
produced were so good why then do we not see navigation and control
devices that can do more that promise a list of a few hundred two
and three word commands. These do not allow sentences internal
clauses or complex phrases. For all the work in the other ares, why
is the NLP community ignoring the fact that companies like IBM and
Dragon are battling for control of a market without even basic NLP
tools to help them along. Download the MemoMaster at our web site
to see a dramatic increase in navigation and control abilities. At VSMM98
a multi-media conference in Japan, I won "Best Technical Award" for
the tools we presented and this was in competition against the U.S.
Navy and Stanford Research Institute's Nuance Communicatons. If
my posts were empty rhetoric one of those two would have won the
prize. In addition we have demonstrated our tools at conferences
and for goverment agencies and at universities. We are currently
working with 4 universities, several companies, and have recently
completed a Department of Commerce Phase I SBIR grant.

>As far as control applications are concerned, the ATIS experiments and
>related efforts were quite informative and were well designed.
>Notable characteristics of the original ATIS efforts which are lacking
>
>a) the ATIS task was designed in consultation with the researchers and
>the ultimate consumers of the technology.
>
>b) the data and tasks were based on an analysis of a real-world
>problem rather than invented to suit a pre-existing set of software.
>
>c) the evaluation was objective rather than conducted by a small,
>self-appointed, obviously biased and highly bombastic group.

Which is precisely why we include an output file that demonstrates our
abilities on the ATIS sentences in the very wide area of funtionality our
contest calls for. We provide a file that gives a full analysis of all the
ATIS sentences and it is available for anyone to download.

>Finally, lack of interest in your "contest" does not imply lack of
>confidence on the part of members of the NLP community. Most of the
>researchers you are trying to taunt do not plan their research efforts
>to try to impress you. My guess based on knowing a number of them is
>that most of them have long since out-grown this sort of double-dare
>and too-scared-to-try sort of rhetoric if not by the end of third
>grade, then at least by the end of their post-graduate studies.

I am not sure we want to frame this in a "too-scared-to try" kind of
a scenario. I think a more appropriate scenario would be something
like the hard sciences where one research group (be it academic or
industrial) is producing results that others cannot. In other fields this
would be taken very seriously. It is only in NLP and linguistics that
researchers feel they can get away with dismissing significant evidence
rather than addressing it. A cry of slander would never be allowed to
pass as counterevidence in the hard sciences. A cry of slander should
be cause for thinking professionals to look closely at the tools we
are providing and then ask "where can an equivalent set be found so
that we do not have to admit that two guys in a small company in Hawaii
are doing far better than major universities and major companies who
have dozens more employees and millions more dollars."

>The fact is, the deafening lack of response that you are noting is an
>indication that nobody feels compelled to provide you with free
>consulting efforts and education in return for uninformed slander.

And of course, you are the one using the term slander. Do you have
anything besides charged rhetoric to back up your claims? Can you
find anyone anywhere who can come even close to providing the
sorts of tools we offer in this very important domain of NLP?

To all the readers of this message, let me make one last statement,
all that Ted Dunning has done is excused himself and others from
demonstrating that they can indeed meet or beat the challenge that has
been issued in the form of this first annual parsing contest. I have
made an empirical claim and have provided factual evidence to support
it on our website at http://www.ergo-ling.com. Can we in academia allow
one group of people to ignore a body of data and clear evidence merely
because they have stated they are above response to it? I believe Ted
Dunning has only further supported my claim that they do not have the
tools or the programs or anything other than promises and dismissive
rhetoric to discount my claims. Our evidence is at the web site, theirs
is yet to be seen.

We won the Best Technical Award at VSMM 98 and expect to do well
elsewhere as well. We do not participate in MUC and TREC currently
because we are working in areas where more sophisticated NLP (rather than
keyword search) is required. That is where we are working and where
we have felt it was necessary and proper to put forward a stricter and
more task appropriate standards and measurements in the form of our
parsing contest. And that is also why we provide an output file of our
analysis
of the ATIS sentences on our web site for all to see. Where are Ted's
and how do they compare. Please check for yourselves our contest
standards are realistic and tough, the sentences are fair and we are
the only company that can produce these results.

Phil Bralich