Topic Modeling, Text Analysis

[gleporeNARA]

Mallet:
http://mallet.cs.umass.edu/index.php

"MALLET is a Java-based package for statistical natural language processing, document classification, clustering, topic modeling, information extraction, and other machine learning applications to text."

see also:
http://programminghistorian.org/lessons/topic-modeling-and-mallet

use standalone Java version at:
https://github.com/senderle/topic-modeling-tool

The below output used 15 as number of topics, and prints the top 5 keywords for each topic. The corpus was a collection of 450 Civil War obituaries, and 50 running race reports (two very different categories of data.)

#################OUTPUT#############
List of Topics

0. Obituaries Russell County Death Date
1. Alderson man Fields county Captain
2. years Mr Rev home church
3. Image Lebanon VA News 11
4. Jones grandchildren Lewis great Browning
5. mile race run Dwight Race
6. Jackson Duff Bundy Steele left
7. Kiser Ball Hurt Hendricks Norton
8. Honaker Fogleman ago Love Buckles
9. death good God friends friend
10. Litton Vicars Vermillion Gap Va
11. Bausell Bays Hill Camp Webb
12. train Porter head man struck
13. race miles time finish running
14. Obituaries Russell County Obituary Soldiers
    ##########################
    Topics 5 and 13 are clearly the 50 running documents, and the other topics neatly highlight various aspects of the Civil War obituaries. We can massage this data by replacing the topic numbers with user-chosen categories:

List of Topics

Obituaries - Obituaries Russell County Death Date
Names, locations, and ranks - Alderson man Fields county Captain
Religious - years Mr Rev home church
Newspaper images - Image Lebanon VA News 11
Family - Jones grandchildren Lewis great Browning
Running and people - mile race run Dwight Race
Family names - Jackson Duff Bundy Steele left
Family names - Kiser Ball Hurt Hendricks Norton
Names and locations - Honaker Fogleman ago Love Buckles
Religious - death good God friends friend
Names and Locations - Litton Vicars Vermillion Gap Va
Names and Locations - Bausell Bays Hill Camp Webb
Death and names - train Porter head man struck
Running - race miles time finish running
Obituaries - Obituaries Russell County Obituary Soldiers

From this we can further group the data into topics, and simplify:

Category 1 - Obituaries, Names, locations, ranks, religious, newspaper images, family, family names, death.

Category 2 - Running and people.

Thus we have gone from close to 500 documents, to two categories in a few minutes.