Log - melsk125/ner GitHub Wiki

This page is the log of this project.

December 2, 2012

Updated yesterday's hack.

Mostly modified corresponding.je.py as written in commit comment. The followings are not-lowered results.

On dataset without hand-edited entries:

  51768 x
   7294 xx
   2578 xxx
    998 xxxx
    426 xxxxx
    221 xxxxxx
    124 xxxxxxx
     60 xxxxxxxx
     39 xxxxxxxxx
     18 xxxxxxxxxx
     14 xxxxxxxxxxx
     10 xxxxxxxxxxxx
     14 xxxxxxxxxxxxx
      2 xxxxxxxxxxxxxx
      2 xxxxxxxxxxxxxxx
      3 xxxxxxxxxxxxxxxx
      3 xxxxxxxxxxxxxxxxx
      2 xxxxxxxxxxxxxxxxxx
      1 xxxxxxxxxxxxxxxxxxxx
      1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

On dataset with hand-edited entries:

 165074 x
  25130 xx
   7226 xxx
   2833 xxxx
   1149 xxxxx
    568 xxxxxx
    309 xxxxxxx
    149 xxxxxxxx
     85 xxxxxxxxx
     49 xxxxxxxxxx
     23 xxxxxxxxxxx
     17 xxxxxxxxxxxx
     15 xxxxxxxxxxxxx
     11 xxxxxxxxxxxxxx
      3 xxxxxxxxxxxxxxx
      2 xxxxxxxxxxxxxxxx
      4 xxxxxxxxxxxxxxxxx
      3 xxxxxxxxxxxxxxxxxx
      1 xxxxxxxxxxxxxxxxxxxx
      1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

December 1, 2012

Created the repository on GitHub.

Did the hack on kw_trans to see the statistics. kw_trans includes data from many sources, both hand-edited and automatically retrieved. The original hack extracts entries automatically retrieved with frequencies 5 or more. This is the result:

 57870 x
 4388 xx
  886 xxx
  251 xxxx
  102 xxxxx
   43 xxxxxx
   13 xxxxxxx
    7 xxxxxxxx
    8 xxxxxxxxx
    5 xxxxxxxxxx
    3 xxxxxxxxxxx
    1 xxxxxxxxxxxx
    1 xxxxxxxxxxxxxxxxxxxxxxxxx

Another similar hack is also run; it also accounts the entries hand-edited. Here is the result:

170182 x
23571 xx
5723 xxx
1910 xxxx
 707 xxxxx
 310 xxxxxx
 119 xxxxxxx
  65 xxxxxxxx
  33 xxxxxxxxx
  16 xxxxxxxxxx
   9 xxxxxxxxxxx
   6 xxxxxxxxxxxx
   1 xxxxxxxxxxxxxxxxxxxxxxxxx