Ted Dunning's answer is presumably that the wordlists are deficient
because they are drawn from a corpus which is not appropriate for the
task/purpose that the user has in hand.
I share the intuitive feeling that there is some validity in the
notion that there are frequent, core words of English at one end of a
scale and other rare, specialist, peculiar words at the other end --
and I think it valuable research to attempt to model this by
collecting large corpora and studying lexical patterns. What is surely
needed is <emphasis>more research</emphasis> on word frequency in
relation to text types, corpus size, sampling procedures, etc. I would
bet 10 pounds that the most frequent word of the British National
Corpus is the same as that in the Bank of English and in the MAP
corpus. But does the rank-ordering of words remain stable through the
wordlist? If it does then we may be getting close to this core
vocabulary -- and if not then there is no option but to build specific
corpora for each application area and task.