Re: Corpora: log likelihood statistic

Ted Pedersen (pedersen@seas.smu.edu)
Tue, 22 Jul 1997 14:07:57 -0500 (CDT)

> >>>>> "dr" == Daniel Ridings <ridings@svenska.gu.se> writes:
>
> dr> Ted Dunning writes about this in "Accurate Methods for the
> dr> Statistics of Surprise and Coincidence" in Computational
> dr> Linguistics Vol. 19, 1993.
>
> dr> Could some kind soul walk me through the formula at the very
> dr> end of the article, using the first bigram in Table 2 for
> dr> illustration? The copy I have in front of me is a reprint in
> dr> "Using Large Corpora" and there appears to be a misprint, so I
> dr> won't repeat the formula here. (The misprint is in no way
> dr> responsible for my dim wit).
>
> dr> Daniel Ridings Gothenburg, Sweden

Ted Dunning has already provided a very useful reply to this question.
I can't add to his discussion, although I'd just like to make a book
recommendation for anyone interested in these sorts of tests:

@book{ReadC88,
author={Read, T. and Cressie, N.},
title={Goodness of fit Statistics for Discrete Multivariate Data},
year = {1988},
address = {New York, NY},
publisher = {Springer-Verlag}}

This book gets into the whole controvery of Pearson's X^2 test vs the
log--likelihood ratio (as did Ted Dunning in CL). It also talks about
exact tests as alternatives to the asymptotic tests and gives some
guidance as to when you should expect certain tests to be valid and when
they might not be.

Some of the same issues are also discussed in this earlier article:

@article{CressieR84,
author = {Cressie, N. and Read, T.},
title = {Multinomial Goodness of Fit Tests},
journal = {Journal of the Royal Statistics Society Series B},
volume = {46},
year = {1984},
pages = {440-464}}

The book is of course more detailed but both are very nice sources.

Best regards,
Ted

-- 
* Ted Pedersen                     pedersen@seas.smu.edu              * 
*                                  http://www.seas.smu.edu/~pedersen/ *
* Department of Computer Science and Engineering,                     *
* Southern Methodist University, Dallas, TX 75275      (214) 768-3712 *