There is a simple reason why EAGLES does not mention the criteria
espoused by Dr Bralich: EAGLES has not so far concerned itself
with proposing standards in the area of parsers.
That is the short answer, interested readers please read on.
It is unfair to criticise us for not doing something we had not
included in our (public) programme of work. One may criticise the
initial selection of topics, however. The topics retained were
those where there was wide agreement that some kind of useful
consensus could be obtained in the near term. The set of topics
we actually worked on were furthermore constrained by factors
such as availability of voluntary labour.
We have worked on the following topics of immediate relevance to
the current debate:
* morphosyntactic annotation of text corpora
* syntactic annotation of text corpora
* morphosyntactic description of lexical items
* syntactic subcategorisation of lexical items (verbs)
* comparative survey of implemented computational linguistic formalisms
* linguistic adequacy of CL formalisms
* development of an evaluation framework for NLP products
At present, we are working on, among other topics, semantic
subcategorisation of lexical items, pragmatic annotation of
corpora (text and spoken language) and developing proposals that
complement ISO 9126 from the point of view of NLP products and
quality in use.
As Dr Bralich has found it difficult to find appropriate
discussions in the EAGLES literature and as there are presumably
others who have experienced such difficulty, let me point out a few
areas of relevance, which also indicate the limits of our work.
The introduction to our document on syntactic annotation of
corpora states:
"The scope of this report is syntactic annotation of corpora. At first
glance, a study of such annotation practices is difficult to
distinguish from a study of parsers, parsing, grammars, the
representation of parses, and the formalisms adopted for such
representations. Clearly, the syntactic annotation of corpora has a
close interrelation with parsing (indeed, a major function of a
syntactically annotated corpus is to provide a test-bed or a
training-bed for wide-coverage parsers). This cannot be ignored in
the report: but what we are ultimately interested in is the
parsing schemes in use to date (i.e. the set of symbols used in the
annotation scheme and guidelines for their application), although
how the corpus is parsed (the parsing system) is relevant, albeit
indirectly, to our task."
As we are also working in a multilingual environment, where
different languages have different linguistic representational
traditions and needs, we find that there are issues in practical
application of guidelines for syntactic representation:
"Since the approach to syntactic annotation is to a large
extent influenced by the language to be annotated, our
guidelines do not give any preference either to a phrase structure
annotation or to a dependency annotation. The
phrase structure annotation, however, is in certain ways the more
demanding of the two, which is why this report
covers phrase structure in more detail. This should not be construed,
however, as expressing a preference for phrase structure annotation.
We will propose notations for both approaches."
In their work, the corpus group took into consideration various
projects including UPenn Treebank, ULancaster Treebank and the
SUSANNE corpus, all of which heavily influenced the shape and
content of our proposals.
This same concern with linguistic representation is met with
throughout the EAGLES reports. Here is an extract from the
document on syntactic subcategorisation, for example:
"The most important concern for EAGLES is linguistic substance.
Consequently, the group is building on the results of the ET-7
feasibility study (Heid & McNaught, 1991) which recommended the
following methodology: to break up the complex descriptive devices
into `minimal observable facts' in order to arrive at the
most fine-grained, common set of features underlying different
theoretical frameworks or systems. EAGLES results are therefore
based on a careful and detailed analysis of different linguistic
theories and frameworks, but aiming at reaching a consensus at
the level of these `minimal observable facts'.
Connected with this basic objective is the approach chosen towards
its achievement, an approach which can be defined as looking for an
edited union (a term due to Gazdar) of the features proposed in the
various major theories and systems. This approach tries to capture
all the relevant distinctions made by the major lexical
theories/systems, without taking a theoretical stand, thereby
giving to features labels which are as neutral as possible.
In an attempt to be as theory-compatible as possible, there are a few
points where choices were left open, especially for those aspects of
grammatical description which tend to be more theory-bound (e.g.
grammatical relations and control). There are practical drawbacks
to this decision -- especially with regard to the implementation of
the proposed standard -- but, at least in this first phase, more
importance was given to avoiding committment to specific theories of
lexical description. We recognise that there is a tension between the
decision to be flexible and open to more than one choice and the real
and effective useability of the proposed standards. Without abandoning
the principle of flexibility and openness, we provide an
indication of usage by exemplifying the implementation of critical
choices.
In general, the EAGLES results are achieved in a dynamic way, with a
cyclical process of revision after one or more phases of
testing and feedback, possibly in large projects. The difference
between the European approach and other approaches to standards
should be pointed out here, to be taken as a description of a
general tendency. While in, say, the USA, a sort of de facto standard
is somehow made available to the community through the provision of
publicly available data, in Europe we try to arrive at consensually
agreed standards. This implies a considerable effort in trying to
involve the relevant experts in the different areas of concern,
either in the phase of producing the standards, or at least in the
successive phases of testing the proposals and providing feedback.
This approach also involves a large amount of overheads in terms of
activities and work necessary to arrive at a consensus as
well as a slower process of arriving at the aimed-for results."
I also draw attention to preparatory work carried out on
computational linguistic formalisms. In particular, the group
charged with this work organised two workshops that brought
together numerous practioners from industry and academia. One
concentrated on intensive comparison of implemented formalisms: a
common framework for comparison was agreed on and systems were put
through their paces as part of the information gathering process.
The other focussed on the linguistic adequacy of implemented
formalisms. I stress that this work was preparatory and was not
continued in phase II of EAGLES, for reasons of little interest in
this context. However, the results did reveal a great degree of
convergence among formalisms and gave indications how grammars
associated with some formalism could be rendered reusable by
another.
The work on evaluation did not specifically include parsers: it
was oriented at developing a general framework for evaluation of
NLP products and focussed initially on adequacy evaluation. There
has been increasing cooperation between EAGLES and those
responsible for ISO 9126, as EAGLES has been instrumental in
providing guidelines that neatly complement the ISO work.
The recent book on NLP evaluation by Sparck Jones & Galliers
recognised the contribution of EAGLES to evaluation.
Lastly, I note that numerous projects and initiatives in Europe
have chosen to adopt EAGLES guidelines especially for the
representation and annotation of text corpora and dictionaries.
ELRA also works with EAGLES guidelines. The initiative thus
receives constant feedback from such widespread take-up of its
results. (EAGLES also has an important activity in the
development of guidelines for spoken language resources and
processing and the speech community has responded warmly to our
efforts in this area.).
Those who wish to find out more about how EAGLES guidelines are
being used, debated, developed, applied, etc., are welcome to
attend (or to acquire the proceedings of) the First International
Conference on Language Resources and Evaluation (Granada, May
1998) where many papers and workshops will refer to EAGLES results
and in a constructively critical way into the bargain.
(Conference URL: http://www.icp.inpg.fr/ELRA/conflre.html)
The EAGLES initiative will produce a new round of publications in
the 4th quarter of 1998, which will be available from
http://www.ilc.pi.cnr.it/EAGLES/home.html
I trust this explanation has served to put EAGLES work and
results in context. One may naturally disagree with our approach;
no-one is imposing standards on anyone in EAGLES. However, we are
encouraged that large numbers of people from industry and academia
have become involved in this initiative and have given freely of
their time to develop recommendations and guidelines that, by most
accounts, meet with widespread approval and adoption. We must be
doing something right...
JMcN
--John McNaught jock@ccl.umist.ac.uk Centre for Computational Linguistics UMIST PO Box 88 Sackville Street Manchester, UK tel: +44.161.200.3098 (direct) M60 1QD fax: +44.161.200.3099