Page Index and Alignment - ZornTaov/ZC-ComicUpdater GitHub Wiki
How chapters.py records which file came from which page (the index), lines it up with the files
(alignment), and repairs skipped, lost, misnamed and unnumbered pages. Chapters depend on it.
A comic scraped from page one has one; others need one built:
python chapters.py index "D:/Comics/Uncompressed/MyComic" # walk the comic, then line it up
python chapters.py align "D:/Comics/Uncompressed/MyComic" # line up a walk already done
python chapters.py show "D:/Comics/Uncompressed/MyComic" # what the last alignment saysThe walk follows the comic from page one recording each page's address, image and title, and downloads
nothing. It writes as it goes, so a stopped walk resumes. --start URL says where the comic begins;
--first follows the comic's first-page link when the metadata only knows a later page. If no
first-page path matches, it stops without writing, since an index begun mid-comic numbers
every page wrong. The index is kept under config/index/ and can be deleted at any time; a walk rebuilds it.
python chapters.py index "D:/Comics/Uncompressed/MyComic" --wordpress --start "https://example.com/comic/some-page/"WordPress's /wp-json/ lists every post of the comic's type with address and featured image, a hundred at a
time: eight thousand pages in minutes instead of a day. --start names any page, and only its post type is
listed (/comic/, not /blog/). A post with no featured image is read for the first image in its body. An
existing index is kept unless --restart sets it aside. Two posts with the same image (/comic/page-40/,
/comic/page-40-2/) are one strip; the first is kept. The list follows post-date order, as a theme's next
link does, but a page linked from nowhere or an image with no post must be added by hand (below);
alignment's size check shows whether list and folder agree. On 429 or 503 requests wait for
Retry-After (or longer each time), and sizes are saved every thousand pages, so a stopped run keeps them.
Files are matched to pages by:
- Filenames, for pages never renamed.
- Image sizes, from HEAD requests that download nothing. Bytes on disk are what the site sent, so a unique size identifies a long-renamed file; sizes that repeat across the comic are matched again within each stretch between pinned pages, where they are almost always unique.
- Counting, for stretches between anchors with equal numbers of pages and files.
Then two checks:
-
Wrong image saved: a file not named after the image the walk saw (a banner or icon caught instead),
reported only when most files do match.
chapters.py refetch <folder> --page N --as-namedfetches the real image under the site's current name and removes the wrong file. - Sizes: a different size alone proves nothing (a moved host re-encodes), but a size belonging to another file in the folder is a conflict.
Alignment must settle before chaptering or renumbering. Pages missing from the folder are listed but block nothing; files that fit no page, and size conflicts, do block it.
Pages the folder lost (deleted, or never saved because something else was saved in their place):
chapters.py refetch <folder> --missing fetches every page the walk has and the folder doesn't, named as a
scrape names them (0191_lojban.png), with the page as the referrer, and never writes over a file already
here. --dry-run first lists them. Line the comic up again afterwards.
A page whose comic isn't one picture (a game, a grid of separate animations) can be kept as a link note of
its address instead, 0042_link.url, which archives draw as a page saying where it is and the Reader
opens:
python chapters.py note "D:/Comics/Uncompressed/MyComic" --url "https://example.com/comic/42/"It is recorded as notes in the comic's metadata, so every scrape and walk keeps that page as one note from
then on. If the walk counted it as several pages, its lines are folded into one and the pages after it
counted on from there (the record as it was is kept beside it as .before-note). Then line the comic up and
refetch --missing writes the note. --forget takes it back. A page whose comic area holds no picture at
all is kept as a note without asking (see Notes and Limits).
When the archive lists 501 pages and the next links visit 500, insert the missing one by hand:
python chapters.py insert "D:/Comics/Uncompressed/MyComic" --dry-run --url "https://example.com/comic/the-skipped-page" --after "https://example.com/comic/the-one-before"--after takes an address or page number. The image is fetched before anything moves, so failure
changes nothing; then later files shift along one, the index learns the page, and alignment re-runs to prove
it. Later chapters start one page later and are flagged for repacking. It needs page-numbered files (run
renumber first otherwise), and in a 500-page comic renames about 400 files, hence --dry-run. Also in
the web page under Chapter boundaries.
A page the site no longer has (the links jump past issue-1-page-25) can only come from elsewhere, such as
the artist's gallery. It matches no walked page, so it reads as a stray and blocks alignment and packing
until declared:
python chapters.py recovered "D:/Comics/Uncompressed/MyComic" --file "page_00026.png" --note "the site's page 25 leads elsewhere; found on the artist's own page"
python chapters.py recovered "D:/Comics/Uncompressed/MyComic" # list what has been put back
python chapters.py recovered "D:/Comics/Uncompressed/MyComic" --file X --forgetName the file so it sorts where it reads; that decides its place. It skips alignment, joins the chapter of the page before it, and keeps its note so nobody has to work out its story again.
When names say nothing (jan.png, then 99002.jpg), a comic this scraper fetched in one pass was
written in reading order, so chapters.py align <folder> --by-time lines up by write time. It needs exactly one file per walked page, else refuses, and the size check afterwards is an independent
verdict. Never use it on files copied, restored from backup, or fetched by several out-of-order runs:
their timestamps mean nothing.
Once alignment is settled, page numbers can be written into the names of a comic scraped without --prefix:
python chapters.py renumber "D:/Comics/Uncompressed/MyComic" --dry-run # say what would change
python chapters.py renumber "D:/Comics/Uncompressed/MyComic"jan.png becomes 0001_jan.png. It refuses unless settled (a guess would be written into the names),
renumbers rather than double-numbers so it is safe to repeat, renames nothing unless every new name is free
and unique, and turns prefix on for new pages. --site-names uses the site's image name instead, as a
--prefix scrape would (0042.png becomes 0043_strip-0043-jan12.jpeg), with the extension from the
file's own bytes. Afterwards run chapters.py repack, or pack for a chaptered comic, since the archive
still holds the old names.