My apologies for the length.
> To the readers:
>
> On March 17th I will be giving a talk at the University of Hawaii's
> Linguistic Department Tuesday Seminar called, "The Best Theory of Syntax."
> In this talk I intend to make the rather non-controversial point that, the
> best theory of syntax must necessarily be the one that demonstrates itself
> to be most completely implemented in a programming language.
I find this point controversial. You say the simplification is justified,
but you make assumptions about why we study syntax, which may not be
justified. See below.
> Some might argue that I am merely putting complex arguments into simple
> language but these arguments have substance and effect in either simple or
> complex langauge. This is especially true when we are dealing with the
> application of syntax to a multi-billion dollar industry such as NLP.
I would propose that your claim be revised: "the best theory of syntax *for
commercial NLP* must necessarily be the one that demonstrates itself to be
most completely implemented in a programming language." I find this version
of the claim uncontroversial.
> More specifically, I intend to present the arguement that the best
> independent and objective measure of a theory of syntax' overall
> effectiveness is its ability to generate, in a computer program, standard
> grammatical structures and to manipulate these structures in the same way
> as users of the language being described.
I am confused on this point. What is a "standard" grammatical structure?
For surely, the grammatical structures used in a theory are
theory-dependent. Your sentence suggests that every parser should be able
to generate similar structures, which would basically ignore the essential
contributions of different theories of syntax. Just compare the
representations used in relational grammar with those of GB; they
demonstrate fundamentally different principles.
I also find it surprising that you claim to know how users of a language
manipulate grammatical structures. If we knew that, we could all go home!
Seriously, I believe this claim is guilty of the same simplification as your
goal of generating "standard" structures; different theories will make
different assumptions about how structures they are representing are
manipulated. Theories of syntax are models, tools for understanding complex
properties of language. I don't think the claim is that these models
represent derivational or representational strategies used by users of
language; rather, they are tools to learn about what kinds of theoretical
principles may account for the use of language.
> That is, I intend to argue that
> the best theory of syntax is the one that produces the best parsers.
If the goal of developing theories of syntax were to develop parsers, then
you could conclude that the best parser (given your measures below)
represented the best theory of syntax, assuming that all theories of syntax
had devoted equal resources to developing parsers. I do not agree with the
implicit assumption that the goal of developing theories of syntax is to
develop parsers. I would propose that the goal of linguistics is to gain
insight into the human mind, a mind which has as one of its main
distinguishing features the ability to learn and use language. One possible
application of this is for parsers, but I think there are plausible reasons
for pursuing both applied and theoretical research in linguistics.
> Following that I will present a very ordinary set of standards for the
> evaluation of parsers and then based on the comparison of theories using
> those standards, I will argue that the theory of syntax that underlies the
> Ergo Linguistic Technologies' parser is the best theory of syntax and that
> all others should be relegated to the scrap heap of "wannabe" theories
> until such time as they can produce equal or better parsers.
As the "President and CEO" of Ergo Linguistic Technologies, your motives
become clear. This no longer sounds like a linguistic argument, but a
commercial.
> The logic
> that I will present to support this is:
>
> 1) if there is ever to be a way to determine which of the
> competing, extant theories of syntax is preferable to the others,
The assumption that one *can* identify a single, preferable theory of syntax
ignores the fact that different theories are developed to explore different
ways of modeling principles of language. (And not necessarily to explore
different ways of modeling the same principles of language). What I mean is,
you need to use the right tool for the right job. If I want to understand
how relationships between elements may affect structure, I'll study
Perlmutter. If I want to explore the role of the lexicon, I'll study LFG.
If I want to find a theory of syntax that is most amenable to computational
modeling, I'll study Ergo.
Let me state the point in another way. I have a colleague who posits that
theories of syntax, while useful for developing our understanding of
language, are less-useful for developing computational systems. There's an
entire field devoted to exploring the application of mathematical processes
to language/text processing (and many of these folks are on this list), and
researchers in this field will probably argue that these statistical
approaches are equally effective, without requiring any knowledge of
theories of syntax. Does that mean that language users manipulate
linguistic structures based upon statistical properties of text? Not
necessarily. It means that some statistical techniques are well-suited to
developing certain types of NLP applications. Does that mean that
statistics is the best theory of syntax? If your premises are sound, then
you must be prepared to accept this possible conclusion.
> 2) since computers have the ability to represent and execute
> binary algorithms, any theory that is composed of binary algorithms should
> be able to be implemented in a programming language. Thus, any theory of
> syntax that has reached a level of maturity should be able to represent
its
> generalizations in working parsers.
I will not argue this point; I believe every first-year student in
computational linguistics has experienced this insight. The non sequitur is
your claim that the quality of a theory of syntax should be measured by the
performance of a parser. Theories of syntax have other uses, and rejecting
one based upon the performance of a parser would be whimsical. Perhaps
computer technology has not yet advanced to the state where we can
effectively program algorithms that capture principles of language; this
doesn't mean that the models of language are immature, it means the
technology is. To discard all other theories of language on this basis
would be whimsical. (And anyway, I don't think there's much danger that
researchers in this field will utter a collective "eureka" and take up the
cause of Ergo.)
> 3) the degree to which a theory of syntax and its algorithms
> cannot be implemented in a programming language is the degree to which
that
> theory and its algorithms have not been completely or correctly worked out
> and should not be considered a mature enough theory to be included in the
> discussion of which theory is to be preferred.
See above. Perhaps it's your programming language, or algorithms, or other
factors which have nothing to do with theories of syntax. Now, if you mean
"preferred" for text processing, that's one thing. But to make general
conclusions about the "best" theory is pointless.
> 4) the theory which is most thoroughly worked out will
> naturally have the most thorough and comprehensive parsing programs
associated
> with it, and for that reason is to be considered the best theory of syntax
as
> determined by this independent, objective criteria.
Assuming, perhaps, that all theories of syntax had applied equal resources
to developing parsers. But as I said above, other factors may influence the
performance of your parser, which do not bear on the sophistication of a
theory of syntax. You seem to insist that there's a connection between the
performance of a parser and the maturity of a theory of syntax, and I think
this assumption is fundamentally wrong.
> It is important to recognize that EAGLES and the MUC conferences, groups
that
> are charged with the responsibility of developing standards for NLP do not
> mention any of the following criteria and instead limit themselves to
> largely general characteristics of user acceptance or vague categories
such
> as "rejects ungrammatical input" rather than specific proposals detailed
in
> terms of syntactic and grammatical structures and functions that are to be
> rejected or accepted.
...
> If the EAGLES' standards are ever to gain any credibility and respect they
are
> going to have to be far more specific about grammatical and syntactic
> phenomena
> that a system can and cannot support.
I'm no expert on EAGLES, but a quick visit to their site tells me that their
standards are for developing components that can be reusable and compatible.
One part of this is evaluation, but it seems to me that their standards
*encourage* different theories and methods of implementation, and do not try
to find the one single "best" one. Since you are trying to use the
standards in a way inconsistent with their stated goals, I am not surprised
you find them inadequate.
> THE CONCLUSIONS I WILL DRAW FROM THIS ARE:
> 1) the theory that underlies the software at Ergo Linguistic
> Technologies
> is not only the best theory of syntax, but is the ONLY theory of syntax
that
> has reached a sufficiently developed state to even attempt the standards
> described here.
I reject the claim, as I described above, that the best parser represents
the best theory of syntax. (Recall, statistics is not a theory of syntax.)
I would accept the claim that the best parser is the best parser, and the
standards you describe above may be useful as one way of describing "best."
> 2) those who do not mention this theory in their research
> proposals, grant
> applications, publications and so on are guilty of negligence (and could
be
> sued if there are grants, contracts, jobs, or other such items of material
> value at stake and where the offerer of these jobs, grants, etc has reason
> to expect that the applicant is an expert in his field and is providing an
> accurate picture of the competitive environment).
Ohmigod, I better call my lawyer! I am under no legal obligation to
recommend any particular system. If I propose that an NLP system is
appropriate for a task, and an offerer of grants, contracts, jobs or other
items of material value agrees enough to grant these items, then there's no
legal negligence. My Ph.D. is an item of material value; if I don't mention
Ergo in my dissertation, are you going to sue me? My dissertation is about
linguistics, not parsers. Market your system on the basis of its commercial
viability and suitability for NLP tasks; don't whine if linguists do not
accept your theories. If you introduce yourself with such confrontational
language, people will confront you!
> 3) All current theories of syntax such as Chomsky's latest or even
> older versions of his theory HPSG, LFG, etc. should all be relegated to
the
> scrap heap of "wannabe" systems until such time as they have been worked
out
> in sufficient detail to allow the creation of programs that can execute
their
> algorithms to the degree required by the above standards.
I cannot even respond to such remarkable statements without serious sarcasm.
Rather than becoming insulting, I'll just make it clear that I find this
suggestion comical.
> Phil Bralich
>
> P.S. We recommend that you download these tools
Only the BracketDoctor and ESE tools are available there. Will you make the
rest of the downloads available?
> P.P.S. As the field of linguistics is dominated by very intelligent, very
> informed individuals who are also quite competitive, you can measure
> the success of this argument on the field overall by the reactions of the
> readers to this post--the smaller the response, the higher the acceptance
> (begrudging though it may be). That is, people are certainly willing to
> criticize any argument they can, but they merely keep quiet if they
cannot.
> Praise for a competitor's arguments is not likely. Thus, a lack of
> criticism should be interpreted as acceptance of these arguments.
I'm compelled to argue with you, just so you don't go claim that I agree
with you!
d'Armond Speers
Ph.D. candidate, Computational Linguistics
Georgetown University
dspeers@thomtech.com