Corpora: Re: "word frequency list"

Gregory Grefenstette (Gregory.Grefenstette@xrce.xerox.com)
Fri, 4 Sep 1998 11:27:13 +0200

> From owner-corpora@lists.uib.no Fri Sep 4 10:49 MET 1998
> From: "Duane Bagaas" <DABuane@sprynet.com>
> Hello
> I understand there is actually a "word frequency list" that has
> been published by some telephone company based on regular
> conversation. Have you seen such a list? I may be looking for the
> “Longman list” if you’re familiar with it. Just the top 800 words.
> Am looking for this because I want to train my voice-recognition
> program quickly.
> Any help you can offer I will greatly appreciate.
> Thank you!
>
> Duane A. Bagaas

Adam Kilgarriff, Senior Research Fellow at the ITRI in Brighton,
(http://www.itri.brighton.ac.uk/~Adam.Kilgarriff/) has created
frequency lists of words from the British National Corpus(BNC).
The BNC is described at http://info.ox.ac.uk/bnc/

Kilgarriff has made these lists available at their ftp site:
ftp://ftp.itri.bton.ac.uk/pub/bnc/README.

in the BNC, given in three columns:
(frequency in BNC) (word) (number of documents)

There are 89740556 word tokens in the BNC (upper and lowercase merged)
and a total of 3209 documents. So a word like

33060 school 2167

appears 33060 times (either as school, School, or SCHOOL)
in the BNC, appears in 2167 (varying size) documents.

89740556 !!ANY 3209
5776487 the 3209
2789557 of 3209
2421353 and 3209
2332421 to 3206
1957310 a 3205
1746903 in 3208
893368 is 3201
891500 that 3175
839971 was 3123
835211 it 3188
786752 for 3200
645364 on 3199
607223 's 3164
606035 with 3197
605764 he 3014
604513 be 3199
594711 i 2834
495810 by 3190
489602 as 3127
435574 at 3179
426213 you 2786
422562 are 3179
419460 his 3002
415981 had 3016
414079 not 3171
404752 this 3184
395999 have 3153
390876 from 3191
390135 but 3168
349129 which 3170
337345 she 2511
336599 they 3137
335983 or 3153
324185 an 3173
312022 her 2544
294309 were 3069
255715 there 3128
249636 we 3029
247596 their 3145
247131 been 3116
242860 has 3156
234595 will 3137
233643 one 3118
222329 all 3161
221172 would 3080
206150 n't 2448
199042 can 3139
189926 if 3107
188141 who 3094
185513 more 3091
185369 when 3061
181647 said 2568
180772 do 2975
173582 what 3055
161205 about 3049
159594 its 3103
155884 so 3050
153528 up 3040
152395 into 3112
151193 no 3074
151029 him 2579
150708 some 3111
148165 could 3039
140989 them 3063
139174 only 3086
136057 time 3085
130808 out 3075
128980 my 2448
127987 two 3092
127882 other 3098
124672 then 3017
123002 may 2984
121995 over 2941
119113 also 3091
118687 new 3067
115477 like 2846
112278 these 3062
111245 me 2287
110111 after 2955
108775 first 3090
108669 your 2443
108462 did 2842
108301 now 3077
108161 any 3060
102750 people 2911
101678 than 3056
99069 should 3025
97958 very 2976
93987 most 2981
92806 see 2780
92635 where 2985
89643 just 2942
88794 made 2971
87126 between 3023
86416 back 2817
83233 way 3010
82920 many 3029
82918 years 3002
82288 being 3029
81997 our 2898
81957 how 2949
80797 work 2834
80326 before 2881
79738 much 2920
78907 down 2850
77238 those 3039
76067 through 2993
73237 such 2923
69260 make 2971
67263 even 2911
67137 us 2705
66792 still 2959
66450 must 2903
66366 own 2976
65715 know 2459
65697 both 2904
65446 year 2861
64266 good 2902
64034 because 2899
63637 too 2886
63625 last 2943
63560 get 2582
61950 three 3006
61521 well 2902
59776 take 2927
59568 however 2854
59270 go 2678
59206 government 2036
58189 under 2963
57966 off 2517
57829 man 2401
57166 use 2620
56456 world 2802
56415 used 2727
54686 same 2967
54463 day 2802
54142 while 2892
53587 life 2792
53540 against 2828
53397 might 2786
53380 right 2102
52934 here 2708
52654 does 2879
51546 another 2943
51504 come 2618
50521 think 2222
49110 old 2721
48635 never 2688
48418 'd 1324
48279 each 2911
48171 since 2668
47744 thought 2071
47551 again 2783
47230 home 2558
46756 need 2437
46750 found 2573
46712 part 2969
46530 mr 1749
46401 say 2476
46145 long 2634
46109 house 2362
45950 ' 2900
44796 place 2822
44264 end 2883
43939 out_of 2840
43852 number 2869
43807 without 2920
43797 going 2621
43604 within 2836
43343 left 2284
43181 different 2834
43021 local 2632
42804 children 2303
42709 system 2505
42611 put 2351
42450 set 2468
42432 during 2837
42259 came 2616
42177 small 2821
41974 although 2797
41832 around 2442
41512 great 2744
41066 got 1815
41061 social 2044
40922 little 2343
40783 'll 1617
40746 case 2674
40361 look 2369
40030 always 2723
40023 've 1708
39729 'm 1535
39366 're 1686
39344 given 2886
39328 why 2604
39079 want 2202
39072 something 2556
38948 group 2684
38824 went 2420
38531 until 2652
38477 company 2298
38445 second 2827
38376 1 2375
37949 party 2152
37912 state 2385
37504 women 2083
37491 per_cent 1568
37289 later 2706
36970 next 2859
36911 high 2816
36802 find 2693
36562 men 2204
36522 information 2519
36361 important 2784
36339 give 2803
36262 public 2383
36183 took 2623
36014 national 2274
35992 head 2392
35821 often 2619
35379 every 2846
34674 help 2606
34603 further 2661
34370 2 2295
34332 british 2324
33881 away 1921
33786 london 2433
33779 seen 2723
33700 less 2629
33673 four 2838
33526 face 2159
33474 told 1911
33420 business 2476
33324 point 2696
33148 taken 2835
33060 school 2167
33009 looked 1563
32940 hand 2592
32865 form 2521
32718 family 2500
32673 general 2194
32653 possible 2795
32567 large 2747
32432 early 2499
32263 best 2678
32179 water 2128
32095 says 2240
31926 fact 2732
31706 night 2112
31704 young 2367
31614 such_as 2564
31527 already 2839
31447 area 2568
31161 asked 2104
31000 things 2501
30939 development 2238
30759 money 2497
30708 better 2560
30672 power 2463
30587 having 2784
30563 yet 2734
30432 perhaps 2567
30360 almost 2672
30281 quite 2562
30183 himself 2214
30173 really 2410
30135 john 2160
30084 though 2415
29888 nothing 2485
29857 called 2510
29827 political 1632
29785 country 2558
29765 days 2663
29651 side 2503
29233 become 2508
29209 change 2459
29046 eyes 1857
28888 whether 2667
28800 enough 2508
28626 market 2040
28504 room 2200
28473 service 2437
28117 far 2639
27960 control 2318
27797 open 2478
27774 support 2215
27771 court 1662
27723 major 2374
27353 members 2375
27321 towards 2652
27171 held 2282
27162 five 2739
26946 others 2706
26945 council 1849
26914 felt 1723
26890 able 2766
26620 times 2731
26612 act 1954
26443 whole 2065
26348 research 1924
26288 following 2334
26220 interest 2645
26120 war 2012
26114 law 1907
26106 available 2539
25924 police 1546
25807 done 2684
25751 working 2492
25671 week 2456
25653 full 2812
25386 problem 2559
25382 making 2783
25331 of_course 2479
25309 either 2468
25263 labour 1128
25210 problems 2494
25156 report 1859
25094 above 2326
25000 once 2426
24998 3 2190
24852 round 1551
24831 level 2360
24827 known 2603
24809 education 1752
24786 voice 1679
24533 saw 2275
24504 up_to 2711
24459 show 2273
24401 body 2357
24339 knew 1850
24334 mother 1726
24301 britain 1942
24225 mind 2230
24100 am 2113
24010 at_least 2685
23941 office 2363
23923 road 1983
23914 car 1896
23864 half 2509
23835 tell 1959
23789 clear 2581
23659 policy 1771
23640 including 2615
23632 minister 1164
23625 turned 1741
23494 main 2634
23490 began 2314
23488 services 2033
23420 thing 2271
23342 several 2644
23332 care 1965
23253 past 1825
23237 a_few 2556
23209 black 1997
23194 seemed 1913
23159 period 2319
23139 keep 2461
23101 across 2507
23072 name 2511
23045 child 1868
23040 health 1954
23040 ever 2528
23036 using 2550
23016 economic 1526
22974 white 2090
22921 million 1809
22917 society 2066
22909 question 2458
22728 for_example 2021
22716 door 1765
22631 months 2555
22623 itself 2649
22595 among 2504
22585 england 2059
22574 upon 2327
22566 feel 1938
22503 father 1694
22451 person 2293
22414 book 2336
22410 effect 2430
22392 together 2671
22383 themselves 2580
22341 likely 2493
22314 words 2336
22247 behind 2398
22194 english 1961
22187 looking 2458
21989 let 1911
21888 areas 2162
21879 south 1729
21846 became 2379
21827 process 2248
21825 position 2514
21816 international 1863
21799 anything 2372
21788 centre 2427
21785 community 1914
21778 woman 1768
21756 city 1888
21661 study 1854
21637 therefore 2184
21566 real 2591
21415 gave 2430
21410 provide 2359
21402 six 2616
21280 staff 1999
21269 result 2469
21237 probably 2505
21216 future 2069
21198 view 2498
21183 action 2418
21144 means 2282
21121 today 2461
21060 present 1917
20987 top 1698
20975 hard 2067
20961 kind 2376
20904 love 1434
20809 special 2610
20790 line 2491
20761 yes 1568
20741 management 1747
20629 evidence 2157
20546 mean 2078
20544 age 2350
20487 west 1754
20475 order 2422
20413 particularly 2552
20404 thus 1976
20359 wanted 1958
20268 sense 2322
20256 rather_than 2388
20249 light 1731
20237 north 1686
20197 experience 2318
20192 particular 2336
20112 close 2114
19989 read 1566
19989 4 2143
19941 taking 2694
19935 difficult 2596
19934 death 2069
19876 big 2191
19875 certain 2517
19825 third 2590
19808 run 1987
19793 range 2270
19712 ca 1683
19657 free 2586
19596 european 1532
19516 god 1241
19475 land 1815
19419 sure 1974
19393 church 1513
19392 whose 2554
19378 outside 2296
19299 century 1878
19268 seems 2455
19189 moment 2121
19175 start 1990
19167 rather 2341
19078 idea 2429
19034 industry 1794
18987 human 1747
18928 play 1952
18924 job 2189
18889 doing 2431
18881 brought 2238
18776 late 2008
18752 10 2175
18727 training 1491
18670 trade 1820
18655 table 2013
18584 history 2194
18583 central 2246
18578 meeting 1720
18545 cost 1874
18519 believe 2016
18514 involved 1936
18504 united 1393
18501 so_that 2480
18482 groups 2044
18456 sometimes 2318
18433 turn 2074
18360 needs 1938
18344 changes 2093
18284 mrs 1356
18279 individual 1705
18194 pay 2122
18183 street 1611
18166 air 2140
18143 few 2642
18130 rate 1960
18001 hands 2181
17960 language 1837
17942 team 1768
17818 based 2294
17804 heard 1740
17783 food 2022
17739 matter 2287
17711 similar 2431
17708 programme 1992
17691 data 1194
17687 section 1926
17667 yesterday 1121
17652 usually 2246
17617 lost 2086
17573 building 1891
17555 leave 2152
17543 cases 2082
17542 common 2347
17534 role 2132
17532 move 2107
17523 europe 1893
17521 sir 1367
17505 5 2079
17495 shall 1871
17493 subject 2102
17469 nature 2306
17407 single 2456
17406 course 2342
17401 states 1263
17370 necessary 2437
17318 uk 1443
17291 along 1809
17287 short 2442
17183 try 2064
17154 price 1981
17141 word 2179
17138 indeed 2343
17134 value 2237
17121 ago 2356
17120 authority 1902
17071 companies 1605
17039 committee 1561
17009 class 1954
17000 bank 1701
16953 patients 740
16951 personal 2340
16910 hours 2301
16827 systems 1579
16807 herself 1439
16803 especially 2371
16799 someone 2015
16784 east 1677
16767 department 1865
16752 former 2112
16743 simply 2433
16696 private 2134
16657 town 1841
16629 total 1775
16612 member 2161
16605 french 1362
16554 figure 2023
16529 reason 2452
16472 practice 2087
16397 true 2390
16383 morning 1898
16357 because_of 2530
16314 expected 2061
16232 return 1808
16183 union 1353
16182 countries 1679
16162 near 2044
16035 low 2122
16031 oh 1140
16023 type 2043
16012 required 1779
15993 else 2176
15992 wife 1775
15956 king 1122
15950 foreign 1673
15942 terms 2236
15922 as_well_as 2385
15877 p. 460
15836 call 1811
15822 increase 1499
15807 financial 1861
15786 friend 1788
15786 american 1825
15782 decision 2134
15766 minutes 1900
15765 president 1112
15729 needed 1840
15706 university 1297
15579 approach 2008
15577 earlier 2111
15543 higher 2176
15542 certainly 2367
15535 club 1452
15504 march 1507
15488 cut 1506
15482 recent 2342
15473 seem 1959
15455 provided 1865
15412 trying 2092
15398 date 1748
15382 live 1831
15341 according_to 2353
15330 ground 2284
15324 strong 2499
15324 hope 1647
15309 led 1821
15286 quality 2154
15251 ask 1910
15246 stage 2341
15209 gone 2190
15196 offer 1990
15190 tax 1103
15190 art 1571
15188 parents 1620
15183 issue 2023
15154 force 1902
15071 added 1512
15070 account 2044
15053 makes 2471
15046 soon 2399
15040 production 1754
14986 final 2370
14984 a_little 1673
14973 as_if 1736
14959 award 858
14951 everything 2134
14944 results 1916
14940 paper 2203
14925 1990 1290
14914 b 1358
14893 concerned 2032
14881 lord 1354
14852 conditions 2052
14825 coming 2010
14817 include 1801
14768 started 1586
14765 getting 2178
14738 living 1584
14710 rest 2367
14702 mr. 372
14660 field 2050
14614 shown 2252
14595 chapter 1288
14592 followed 2125
14580 secretary 1478
14572 described 1731
14452 various 2351
14409 royal 1746
14368 despite 2234
14317 project 1559
14285 david 1600
14285 costs 1405
14283 greater 2175
14242 decided 1803
14241 april 1676
14237 schools 1353
14236 clearly 2326
14226 music 1477
14207 knowledge 2121
14154 situation 2225
14135 girl 1365
14104 students 1379
14100 hospital 1421
14082 poor 2199
14066 workers 1456
14051 tried 1533
14036 bed 1513
14008 friends 2030
13991 series 2266
13963 amount 2090
13955 june 1750
13932 sort 1975
13895 below 1932
13886 record 1945
13878 ten 2194
13855 natural 2177
13842 game 1604
13836 lead 1988
13824 ways 2367
13819 remember 1651
13791 talk 1573
13761 agreed 1466
13737 anyone 2162
13709 red 1797
13686 sound 1409
13599 current 1786
13558 &amp 1424
13539 paid 1742
13537 forward 2136
13481 month 2077
13480 studies 1366
13477 met 1761
13470 moved 1660
13464 basis 2084
13451 front 1391
13406 weeks 2224
13403 planning 1238
13399 understand 1974
13378 6 1941
13374 news 1987
13370 green 1300
13368 1989 1298
13357 works 1726
13341 20 1986
13330 security 1768
13319 easy 2181
13297 hair 1278
13277 points 1995
13270 wrong 2075
13255 allowed 2086
13221 inside 1676
13220 feet 1686
13209 bring 2383
13177 plan 1807
13171 manager 1470
13158 attention 2334
13150 died 1488
13145 heart 2095
13141 simple 2284
13139 actually 2272
13135 kept 1833
13112 considered 1906
13067 meet 2280
13067 material 1902
13063 carried 1919
13054 capital 1422
13006 movement 1964
12987 deal 1707
12972 analysis 1443
12964 computer 1320
12953 structure 1721
12945 dark 1237
12938 success 2250
12894 stood 1395
12874 test 1566
12868 de 1356
12857 model 1609
12849 agreement 1734
12846 share 1397
12807 questions 2094
12798 alone 2018
12794 received 1748
12767 board 1496
12763 1991 1185
12762 theory 1521
12759 per 1771
12718 population 1427
12704 evening 1788
12690 story 1983
12675 legal 1605
12656 modern 2003
12655 whom 2172
12639 increased 1292
12618 performance 1866
12606 growth 1622
12601 scotland 1436
12593 continued 1485
12590 running 1969
12571 sent 1756
12568 sea 1528
12567 answer 1927
12561 written 2048
12561 comes 2390
12535 fire 1731
12532 cause 1792
12527 bill 1119
12526 stop 1840
12518 finally 2363
12517 design 1567
12505 produced 1741
12498 rights 1590
12463 at_all 2168
12433 saying 1970
12373 bad 2057
12362 relationship 1895
12359 son 1589
12336 in_order 2157
12330 parties 1654
12292 environment 1741
12275 12 1987
12252 books 1920
12229 visit 1564
12229 letter 1912
12226 reached 1768
12224 authorities 1517
12188 complete 2287
12163 chance 2150
12160 nor 2211
12110 everyone 2072
12105 feeling 1547
12099 hold 1993
12095 press 1670
12089 property 1642
12081 standard 1317
12053 developed 1748
12040 included 1510
12021 st 1516
12013 built 1864
11962 behaviour 1594
11930 prime 1453
11928 france 1588
11893 throughout 2269
11883 lower 2028
11878 treatment 1671
11878 happened 1428
11815 space 2110
11792 key 1565
11782 example 1922
11771 reported 1282
11764 firm 1233
11757 energy 1656
11745 director 1606
11743 previous 2274
11736 serious 2253
11711 established 1613
11679 recently 2344
11664 significant 1842
11643 dead 1736
11635 boy 1436
11626 middle 1522
11615 levels 1743
11609 size 2089
11578 wide 2172
11561 30 1855
11537 normal 2090
11527 giving 2476
11504 peter 1567
11504 july 1576
11493 forces 1451
11482 list 2102
11481 seven 2267
11463 quickly 2207
11458 c 1217
11445 risk 1656
11435 beyond 2260
11419 miss 907
11419 issues 1659
11396 term 1944
11395 technology 1393
11387 soviet 519
11376 sat 1140
11360 direct 1954
11347 choice 2203
11336 details 2181
11323 suddenly 1625
11322 hall 1129
11312 defence 1493
11297 scheme 1668
11295 thinking 1743
11293 beginning 1874
11291 activities 1893
11280 loss 2081
11260 fine 1931
11246 military 1121
11243 contract 1324
11221 pressure 2000
11216 income 1212
11212 hear 1657
11207 industrial 1457
11182 attempt 1904
11180 wall 1620
11170 worked 1647
11167 mark 1150
11165 plans 1854
11150 activity 1807
11145 opened 1357
11116 forms 1581
11114 shows 1719
11105 continue 2108
11098 cup 1088
11089 association 1410
11080 rates 1303
11069 german 1077
11057 leading 1815
11053 hotel 1203
11049 consider 1815
11046 rose 1327
11039 relations 1439
11030 won 1235
11010 eight 2187
10997 spent 1671
10985 parts 2244
10982 dr 1253
10969 15 1925
10965 stay 1721
10965 away_from 2195
10962 army 1330
10956 worth 2060
10952 generally 1979
10940 sun 1473
10937 produce 1940
10914 allow 2124
10894 original 2099
10892 myself 1505
10874 teachers 978
10857 chief 1191
10838 arms 1552
10832 appropriate 1912
10828 potential 1362
10794 professional 1781
10780 cold 1341
10763 claim 1303
10715 specific 1751
10706 village 1332
10699 appeal 1433
10693 basic 1888
10656 7 1897
10650 male 908
10633 suggested 1440
10631 summer 1997
10600 demand 1552
10598 reports 1368
10594 writing 1615
10593 appear 1974
10591 please 1404
10588 season 1147
10563 floor 1668
10547 played 1352
10540 wish 1645
10527 published 1392
10510 takes 2229
10506 product 1528
10504 colour 1723
10502 george 1451
10502 fall 1662
10500 park 1061
10497 appeared 1625
10489 wo 1495
10479 northern 1073
10477 effects 1705
10476 rules 1673
10420 independent 1849
10419 unit 1662
10412 husband 1453
10395 announced 1021
10374 hon. 26
10373 ideas 1850
10359 access 1731
10349 style 1804
10348 investment 1243
10346 successful 2178
10328 showed 2060
10318 figures 1979
10318 1988 1247
10317 popular 1957
10263 meant 1510
10260 garden 1373
10251 october 1680
10248 deep 1527
10246 germany 1417
10243 returned 1299
10240 whatever 2176
10240 interests 1686
10233 rise 1515
10231 1992 1132
10230 fell 1878
10216 events 2050
10210 numbers 1994
10203 paul 1356
10202 products 1441
10201 changed 1941
10198 influence 1564
10189 passed 1701
10177 nearly 2198
10173 science 1263
10173 lot 1413
10159 offered 1743
10145 fish 1123
10126 each_other 2041
10121 couple 1681
10105 cover 1602
10088 limited 1569
10077 reasons 2195
10073 event 2063
10063 economy 1219
10054 reading 1225
10051 commission 1063
10022 more_than 1706
10021 immediately 2201
10005 employment 1193
9940 september 1652
9922 picture 2001
9919 difference 2044
9918 ensure 1845
9914 benefit 1601
9910 circumstances 1925
9908 resources 1583
9906 happy 1905
9900 concern 1844
9880 note 1597
9876 aware 2049
9873 follows 1624
9862 raised 1557
9853 designed 1656
9821 lay 1472
9814 buy 1830
9803 miles 1639
9776 oil 1382
9772 advice 1904
9770 lack 2064
9767 western 1317
9759 win 1304
9753 january 1548

____________________________________________________________________________
Gregory Grefenstette, Multilingual Theory and Technology
Xerox Research Centre Europe
6 chemin de Maupertuis, 38240 Meylan, France
Gregory.Grefenstette@xrce.xerox.com
Phone : (33) 4 76 61 50 82 fax : (33) 4 76 61 50 99
Inside France: 04-76-61-50-82