Introduction - internetarchive/heritrix3 GitHub Wiki

Heritrix is the Internet Archive's open-source, extensible, scalable, archival-quality Web crawler.

This document explains how to install, configure, and use Heritrix to crawl the Web.  It assumes the reader has a general understanding of computing concepts such as HTTP and URIs, 

Audience

The audience of this document is Heritrix administrators and other technical staff who want to crawl the Internet using Heritrix.

Versions

The information in this guide is for Heritrix 3.0 unless otherwise noted.  Sections that provide information about Heritrix 3.1 are marked by an "As of Heritrix 3.1" clause.