-
Notifications
You must be signed in to change notification settings - Fork 760
Force speculative embed URIs into single queue.
Alex Osborne edited this page Jul 4, 2018
·
2 revisions
Use of the following setting will send speculative embeds ('X' hoppath found via extraction from javascript) into a single, separate queue.
This was needed due to the javascript extractor enqueuing a large number of faulty/non-existent URIs.
<!-- extracted uris go to their own queue -->
<bean class='org.archive.crawler.spring.DecideRuledSheetAssociation'>
<property name='rules'>
<bean>
<property name="regex" value=".*X$" />
</bean>
</property>
<property name='targetSheetNames'>
<list>
<value>speculativeSingleQueue</value>
<value>speculativeSingleQueueBalance</value>
</list>
</property>
</bean>
<bean id='speculativeSingleQueue' class='org.archive.spring.Sheet'>
<property name='map'>
<map>
<entry key='queueAssignmentPolicy.forceQueueAssignment' value='speculative-queue' />
</map>
</property>
</bean>
<bean id='speculativeSingleQueueBalance' class='org.archive.spring.Sheet'>
<property name='map'>
<map>
<entry key='frontier.balanceReplenishAmount' value='2000000000' />
</map>
</property>
</bean>
Structured Guides:
User Guide
- Introduction
- New Features in 3.0 and 3.1
- Your First Crawl
- Checkpointing
- Main Console Page
- Profiles
- Heritrix Output
- Common Heritrix Use Cases
- Jobs
- Configuring Jobs and Profiles
- Processing Chains
- Credentials
- Creating Jobs and Profiles
- Outside the User Interface
- A Quick Guide to Creating a Profile
- Job Page
- Frontier
- Spring Framework
- Multiple Machine Crawling
- Heritrix3 on Mac OS X
- Heritrix3 on Windows
- Responsible Crawling
- Politeness parameters
- BeanShell Script For Downloading Video
- crawl manifest
- JVM Options
- Frontier queue budgets
- BeanShell User Notes
- Facebook and Twitter Scroll-down
- Deduping (Duplication Reduction)
- Force speculative embed URIs into single queue.
- Heritrix3 Useful Scripts
- How-To Feed URLs in bulk to a crawler
- MatchesListRegexDecideRule vs NotMatchesListRegexDecideRule
- WARC (Web ARChive)
- When taking a snapshot Heritrix renames crawl.log
- YouTube
- H3 Dev Notes for Crawl Operators
- Development Notes
- Spring Crawl Configuration
- Potential Cleanup-Refactorings
- Future Directions Brainstorming
- Documentation Wishlist
- Web Spam Detection for Heritrix
- Style Guide
- HOWTO Ship a Heritrix Release
- Heritrix in Eclipse