| title | layout |
|---|---|
MeTA |
default |
- text tokenization, including deep semantic features like parse trees
- inverted and forward indexes with compression and various caching strategies
- a collection of ranking functions for searching the indexes
- topic models
- classification algorithms
- graph algorithms
- language models
- CRF implementation (POS-tagging, shallow parsing)
- wrappers for liblinear and libsvm (including libsvm dataset parsers)
- UTF8 support for analysis on various languages
- multithreaded algorithms
See the setup guide for installation instructions.
We have walkthroughs for parts of MeTA listed under the Tutorials header.
Doxygen documentation can be found here.
Visit the MeTA forum to discuss anything surrounding the toolkit. You will find user support topics, community-written documentation, and the developer discussions here.
If you used MeTA in your research, we would greatly appreciate a citation for our ACL demo paper:
{% highlight latex %} @InProceedings{meta-toolkit, author = {Massung, Sean and Geigle, Chase and Zhai, Cheng{X}iang}, {% raw %}title = {{MeTA: A Unified Toolkit for Text Retrieval and Analysis}},{% endraw %} booktitle = {Proceedings of ACL-2016 System Demonstrations}, month = {August}, year = {2016}, address = {Berlin, Germany}, publisher = {Association for Computational Linguistics}, pages = {91--96}, url = {http://anthology.aclweb.org/P16-4016} } {% endhighlight %}
Recent years have seen a dramatic growth of natural language text data, including web pages, news articles, scientific literature, emails, enterprise documents, and social media such as blog articles, forum posts, product reviews, and tweets. This has led to an increasing demand for powerful software tools to help people manage and analyze vast amount of text data effectively and efficiently. Unlike data generated by a computer system or sensors, text data are usually generated directly by humans for humans.
This has two consequences. First, since text data are generated by people, they are especially valuable for discovering knowledge about human opinions and preferences, in addition to many other kinds of knowledge that we encode in text. Second, since text is written for consumption by humans, humans play a critical role in any text data application system, and a text management and analysis system must involve them in the loop of text analysis.
Existing toolkits supporting text management and analysis tend to fall into two categories. The first is search engine toolkits, which are especially suitable for building a search engine application, but tend to have limited support for text analysis/mining functions. Examples include Lucene, Terrier, and Indri/Lemur. The second is text mining or general data mining and machine learning toolkits, which tend to selectively support some text analysis functions, but generally do not support search capability.
However, seamless integration of search engine capabilities with various text analysis functions is necessary due to two reasons. First, while the raw data may be large for any particular problem, it is often a relatively small subset of the data that are relevant, and a search engine is an essential tool for quickly discovering a small subset of relevant text data in a large text collection. Second, search engines are needed to help analysts interpret any patterns discovered in the data by allowing them to examine the relevant original text data to make sense of any discovered pattern. A main design philosophy of MeTA, which also differentiates MeTA from all the existing toolkits, is its emphasis on the tight integration of search capabilities (indeed, text access capabilities in general) with text analysis functions, enabling it to provide full support for building a powerful text analysis application.
Another design philosophy of MeTA is to facilitate education and research experiments with various algorithms. In this direction, it is similar to Indri/Lemur in its emphasis on modularity and extensibility achieved through object-oriented design. It enables flexible configuration of a selected subset of modules so as to make it easy for designing course assignments or experimenting with a few selected algorithms as needed in focused research projects. For example, it has been successfully used in a MOOC on Text Retrieval and Search Engines where over one thousand Coursera learners have used the toolkit to finish a large programming assignment. It will be used again for supporting programming assignments for another upcoming MOOC on Text Mining and Analytics.
For more discussion about how search and text analysis are related and how to build a general text analysis engine, please see the this presentation.
- The Coursera course Text Retrieval and Search Engines uses MeTA in programming assignments available to thousands of students
- The Coursera course Text Mining and Analytics uses MeTA in its programming assignments as well
- An upcoming textbook Text Data Analysis and Management: A Practical Introduction to Text Mining and Information Retrieval showcases the MeTA toolkit with exercises and demos
- The UIUC course CS 410: Text Information Systems uses MeTA in some programming assignments
- The TIMAN Research Group from the UIUC Computer Science Department uses MeTA in their text mining research
Contact us through our GitHub issues page if you'd like your application of MeTA on our site!