-
Notifications
You must be signed in to change notification settings - Fork 760
Heritrix Configuration
Alex Osborne edited this page Jul 6, 2021
·
2 revisions
-
Set the
JAVA_HOME
environment variable. The value should point to the JRE 11 Java installation. For example, this variable might be set to/usr/local/java/jre
export JAVA_HOME=/usr/local/java/jre
-
Set the
HERITRIX_HOME
environment variable. The value should be set to the main Heritrix directory, which is below thebin
directory.export HERITRIX_HOME=/PATH/TO/HERITRIX
For example, an installation of Heritrix with a
bin
directory located at/home/user/heritrix3.1/bin
would be configured as follows.export HERITRIX_HOME=/home/user/heritrix3.1
-
Set execute permission on the Heritirix startup file.
chmod u+x $HERITRIX_HOME/bin/heritrix
-
To change the amount of memory allocated to Heritrix (the Java heap size), set the
JAVA_OPTS
environmental variable. The following example allocates 1GB of memory to Heritrix.export JAVA_OPTS=-Xmx1024M
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse