Notes and Limits

What will not work, the site quirks the scraper handles, and what it leaves to you.

Limits and manners

• Logins are not supported. JavaScript-only pagination will not work as-is; -ej turns JavaScript on (off by default for speed; see When the check and a scrape disagree).
• For an unknown site, add an element path. As last resorts, -m skips the paths and takes the first image on the page (for a page holding only the comic), and -n presses next with a path written by hand into next() in mirror_base.py.
• Be considerate: this drives a real browser against someone's site; -w waits between pages.
• The browser is only asked for element attributes: it loads no images (fetched once, by requests) and does not wait for the load event, so a hanging third-party font or script cannot stall a run.
• When a comic's newest page has different markup (themes often wrap the image in a next link, which the newest page lacks), the path search runs again rather than giving up.

The end of the comic

• Next leads to a page with no comic image (often the front page): the comic has caught up, once a page was saved. The landing address is reported and the resume address stays on the last page saved. No image on the first page is still code 3.
• Next on the newest page wraps to page one: the run ends there, recognised by the index or the address first scraped.
• Next is the page's own address plus # (?comic=500#, ?comic=500#comic): the same page, so caught up, and not recorded by a walk, as long as it shows the same image for a few seconds. A JavaScript comic routing by fragment (#/page/41 to #/page/42) draws a new image and is followed.
• Before calling it caught up because next stays put, every other known next path that points at something else on the page is pressed, and the first to reach a page whose picture the comic does not hold is used from then on. A newest page can carry a teaser for the page to come, linked to itself, which a loose next path takes for the next button; without this, a comic that remembered it sat on that page while new ones came out. A path that leads to a page already held, or back to the start, is not taken. A walk does the same. And a run ending on its newest page remembers the path that last went somewhere new, not whatever it pressed on that last page.
• A scrape never decides a comic has ended. ended is yours, set in the adopt report, the web page or the metadata, and carried through untouched. state.completed only means that run reached the latest page.

Next links that go the wrong way

• Backwards (code 8): an image wrapped in a link to the previous page would re-download the comic backwards; a last-page next pointing at the front looks like a redesign. A run stops if the page after the first is one already held. --no-direction-check turns this off for a comic that genuinely reuses filenames. (One backwards step at a chapter boundary is tolerated; see When to number pages.)
• Two next links (code 10): a site with one next through everything and one through a single series skips pages silently. With an index, a next that overshoots the expected page is retried with every other known next path, and the first to hit it is used from then on; if none does, the run stops before saving past the gap.

An image path that finds the wrong picture

A comic remembers the image path that found its pages. On a page where that path finds nothing:

• It looks again in the same place first. //*[@id="comic"]/img becomes //*[@id="comic"]//img, so a comic wrapped in a link on one page is still found where it always is.
• A page whose comic isn't a picture - one you play with, built by the page's own script - has the container but no picture in it. It is kept as a link note, 1608_link.url, holding the page's address: it keeps its number, archives draw a page saying where it is, and the Reader offers Open link. Nothing else on the page is taken for the comic.
• Only where the container itself is gone is the whole list of paths searched, and a path found that way is used for that page only. It becomes the comic's path once the old one has found nothing three pages running, which is the site changing its layout rather than one odd page. A picture such a path finds that the page before also showed, anywhere on it, is the site's own - a logo, an ad - and is left out. So a newest page that is a teaser locked for subscribers, with nothing on it but the site's header, saves nothing: the comic has caught up for now, and the next run carries on from the last real page.
• The same picture as the comic on 5 pages in a row (code 11) stops the run: a menu drawn as a picture below every comic, say, found by a path meant for another site. The message names the path that found it and the pages already saved as it, to take out, and suggests --check on the page to find the right path.

Filenames

• A page keeps one filename: if the re-saved starting page lands under a new name, the newer wins and the old file leaves the folder and archive (hand-numbered pages, doubled extensions).
• Several pages on one address are each saved and numbered; see Several pages on one address.