Corpus details - Git-Lit/git-lit GitHub Wiki

Overview

The initial git-lit corpus comes from the British Library Labs 19th Century Books collection of 68,000 volumes (25 million pages) published between 1789 and 1914. The 65,000 books represent less than 7% of the British Library's 1+ million book collection, but they are not a representative sample and they're not necessarily first or significant editions.

The Microsoft funded scanning project done in 2006 focused on a single physical storage area containing books collected between 1850 and 1975 in four subject areas: English literature, geography, general history, and philosophy (as those subject areas were defined at the time). They are predominantly in English and published in Britain between 1800 and 1936. Due to scanning limitations, all volumes which were not 18.5-30.5cm tall and less than 10 cm thick were excluded.

http://labs.bl.uk/Digital+Collections+-+Books+and+Text#x19th Century Books

Selection Criteria

It probably makes sense to prioritize work on the BL corpus in phases and some volumes may be so low priority as to not be worth including at all.

Please add, rearrange, discuss!

Higher priority

  • Rare or unique editions
  • Good quality OCR
  • "easy"
  • popular?
  • balanced author gender, nationality?

Lower priority

  • editions which have high quality text available from Project Gutenberg or other sources or which are actively being worked on at another site (e.g. DNB at WikiSource)
  • editions which don't have all volumes scanned, e.g. 002462923 where we have only have 9 volumes (2, 4, 13, 15, 18, 22, 27, 30, 33) of a 20 bound volume set (34 logical volumes)
  • low quality OCR
  • "difficult," whatever that means (or should we not care)

Statistics

(move highlights to main readme)

  • 49,455 editions, comprising 65,235 volumes
  • 43,404 single volume editions
  • 7,149 multi-volume editions with 21,831 volumes total
  • 28 languages (but overwhelmingly English, of course)
  • 25,572 personal authors
  • 810 corporate authors
  • 6,766 editions with no author listed
  • 329 authors with 10 or more editions represented
  • Most pages in single volume - 1,884 - Úplný místopisný slovník Královstvi Českého, etc. (II. Část historická.-Místopisný slovník historický Království Českého. Sepsal A. Sedláčck.)
  • Most volumes - 66 - Dictionary of National Biography. Edited by Leslie Stephen. vol. 1-21. (By Leslie Stephen and Sidney Lee.) vol. 22-26. By Sidney Lee. vol. 27-63. [With Supplementary volumes.]
  • Most pages - 32,072 - also DNB (see above)
  • Fewest pages - 4 (two instances + one with 5 pages)
  • Best represented authors - 154 books Byron, 106 Sir Walter Scott, 103 Mrs. Henry Wood, all other less than 100
  • 19,260 authors with only a single book, 3,471 authors with two books, 1,178 with three books, 547 with four, 309 with five, 489 with 6-10, 245 with 10-20 books, 84 with 20-50 books, and 7 authors with 50 more more books.

Top Languages

(original code + language detection for unknowns)

English	34907	70.6%
French	 3857	 7.8%
German	 3222	 6.5%
Unknown	 3073	 6.2%
Spanish	  757	 1.5%
Italian	  694	 1.4%
Multiple  576	 1.2%
Russian   557	 1.1%
Dutch	  552	 1.1%

Top Authors**

Byron, George Gordon Byron, Baron, 1788-1824	154
Scott, Walter, Sir, 1771-1832	106
Wood, Henry, Mrs., 1814-1887.	103
Oliphant, Mrs. (Margaret), 1828-1897.	74
Dickens, Charles, 1812-1870	66
Shakespeare, William, 1564-1616	61
Marryat, Florence, 1837-1899.	58
Goldsmith, Oliver, 1730?-1774	55
Dryden, John, 1631-1700.	48
Ainsworth, William Harrison, 1805-1882.	47
Burns, Robert, 1759-1796	45
Payn, James, 1830-1898.	42
FENN, George Manville.	41
Norie, J. W. (John William), 1772-1843.	41
CAREY, Rosa Nouchette.	40
Marshall, Emma, 1830-1899	40
Riddell, J. H., Mrs., 1832-1906.	40

Top 10 downloads from JISC Historic Books Aug 2011 - Feb 2013

(per YouTube video from 2013 describing the collection)

  1. Mariana Starke, Information and Directions for Travellers to the Continent
  2. Spenser, Complete Works
  3. Rae, Newfoundland to Manitoba
  4. Blake, Selections from Writings of William Blake
  5. Dictionary of National Biography
  6. A non-military journal, or observations made in Egypt..
  7. Stickney, Women of England, their social duties...
  8. Evans, Letters written during a tour through South Wales
  9. Barrie, City of Dundee illustrated
  10. Bey, Travels of Ali Bey

All languages (original coding)

eng	19250
und	18877
fre	3833
ger	3161
spa	769
ita	642
rus	582
mul	576
dut	551
hun	256
swe	244
dan	231
pol	175
por	75
lat	74
cze	58
gre	45
fin	19
scc	12
gle	5
ice	3
lit	3
rum	3
scr	3
bul	2
nno	2
slo	2
slv	2