Release Notes Heritrix 3.4.0 20190207 - internetarchive/heritrix3 GitHub Wiki
For an overview of this release, see A New Release of Heritrix 3.
The following summarises the changes since late 2017.
- ToeThread death when using HighestUriPrecedenceProvider
- Add checks to guard against server sending 304 in error
- Ensure scope logs have different names per job
- HBase fixes: #222, #243, #224
- Do not checkpoint if crawl job has not started
- Resolved significant BDBFrontier thread safety issues
- HTML extractor does not handle the base href correctly when it's relative
- Catch exceptions scoping outlinks to stop them from derailing process
- Fix for test failures in a workspace on NFS-mounted filesystem
- Enforce robots.txt character limit per char not per line
- Treat a failed fetch (e.g. socket timeout) of robots.txt the same way as http errors
- Allow JavaDNS to be disabled as part of resolving outstanding build and test issues
- WARCLimitEnforcer.java - Add support for multiple warc writers.
- By default only execute decide rules if they might change the outcome NOTE that this is potentially a breaking change IF decide rules have side effects you rely on!
- Can now deploy to Maven Central
- Add a simple way to launch the
latestcheckpoint - Add parameter to allow even distribution for parallel queues.