-
Notifications
You must be signed in to change notification settings - Fork 14
Corpora
Siamak Barzegar edited this page Jul 14, 2017
·
7 revisions
Each pre-build models processed one of the following text corpora.
Available only in English, this corpus was offered Google. For more information, please check the official page.
The preprocess identified multi-word expressions transforming them in one unique token with underscore (Barack Obama -> Barack_Obama).
- remove stop-words: false
- lowercase: false
- minimum word size: 1
- apply stem: false
- apply lemma: false
Available for all languages, this corpus is the dump of the Wikipedia in 2014 (except JA and KO) and 2016 (for JA and KO).
During the preprocess step, we ignored Redirect and Disambiguation pages.
- remove stop-words: true
- lowercase: true
- minimum word size: 3
- apply stem: true
- apply lemma: false
Available only in English, this corpus was offered by:
- ConceptNet 5.5, which contains data from Wiktionary, WordNet, and many contributors to Open Mind Common Sense projects, edited by Rob Speer
- Common Crawl, this corpus was suggested by Jeffrey Pennington, Richard Socher, and Christopher Manning. For more information, please check the official page.
- Google News, this corpus was offered by Google. For more information, please check the official page.