Potential Cleanup Refactorings - internetarchive/heritrix3 GitHub Wiki

• merge small Util classes to fewer larger ones?
• do we need both a org.archive.util.IoUtils and org.archive.crawler.util.IoUtils?
• do we need separate IoUtils and FileUtils
• should the package be shrunk or split to subpackages?
  • move blooms to subpackage?
  • move SURT classes to URI-related package?

CrawlURIs need a flexible generic serializable data structure for arbitrary markup by loosely-coordinated extensions. This has been AList but our discmofort with it has caused us to deprecate still-useful direct accessors.

Should we try another structure constrained to have a convenient serialization format-- perhaps JSON? JSON+Objects? YAML?

• evaluate for refactoring all classes >1000 lines?

• deprecate SupplementaryLinksScoper – subsumed by mappers, main
  LinksScoper's logging?
• reconcile dash-underscore_CamelCase conventions?

In places (specifically around the UURI/CrawlURI classes) we've overridden Object.toString() to return a more 'naked' representation of a object, and then relied on that toString() for functionality.

Unfortunately, given toString()'s special role as default string-ification of any object and use through debugging logs/itnerfaces, this hides useful info – like the class of anything that reports a toString() URI.

I would prefer any meaningful string-version of an object to be accessed via other methods, retaining toString() in its default implementation or some other rich, debugging-centric rendering that can be adjusted fearlessly without impacting application functionality.

OK to use JS, should never be necessary for UI to be usable

allow tallies (and rates?) of generic named quanitites – not only the static set of values defined in interface methods