Frontier

The Frontier is a Spring bean that maintains part of the internal state of the crawl.  Specifically, it stores the URLs that have been discovered but not yet crawled, and is resposible for 'releasing' those URLs to the ToeThreads so they get crawled at the appropriate time and in the appropriate order.

There is only one Frontier per crawl job.

Other systems within Heritrix3 store information on what has been crawled so far:

• The UriUniqFilter filters out URIs before they go into the frontier, and is usually used to ensure URLs get crawled once per crawl. May or may not persist state between crawls.
• Deduping (Duplication Reduction) if enabled, which is responsible for deduplicating WARC content against previous crawls based on payload digest.

There are other parts of the code that use the state system, but these are usually just caches (IP lookups, robots.txt, etc.) rather than being critical crawl status information.

Crucially, the Heritrix3 frontier does not only store multiple queues of URLs to crawl in priority order, it also controls the crawl-delay politeness settings on a per-queue basis. i.e. it controls when CrawlURIs are due to be crawled, not just the crawl priority.

The Heritrix BdbFrontier also implements queue rotation, to ensure all queues are visited even when there are far more queues that available threads to perform the crawl. This means Heritrix crawl queues have 'session' budgets (to handle rotation) as well as overall crawl quotas (which are applied to the whole crawl).

In Heritrix 3.0 and 3.1 there is only one kind of Frontier, the Heritrix BdbFrontier.  Other Frontiers that were included in Heritrix 1.x are no longer supported.

For more details see:

• Frontier Settings
• Frontier queue budgets
• Heritrix BdbFrontier for technical details
• Developing Alternative Frontier Implementations