Archives and ComicInfo

How archives stay cheap to sync, the ComicInfo.xml each carries, and stand-ins for pages no reader shows.

Why appends are cheap

A zip keeps entries in write order with its index at the end, so new pages are written after the existing bytes and only the index is rewritten. On a real 250 MB archive, a fortnight of pages left 97.8% of the file byte-identical in a tenth of a second, so rsync and Syncthing carry only the new pages and ZFS or btrfs snapshots do not grow by a whole archive per update. Pages are stored, not deflated: JPEG and PNG gain nothing from it. Archives are written aside and moved into place, so an interrupted run leaves nothing half-written.

ComicInfo.xml

Every archive (single, per chapter, repacked or adopted) carries a ComicInfo.xml to the v2.1 draft schema: series, title, source site, page count, and each page's size in bytes and pixels. A page much wider than tall is marked a double page, shown alone in a spread; a stand-in (below) is marked as not a story page. Sizes come from each file's first bytes and carry over between ComicInfos, so a page is measured once. Chapter archives add the chapter name, a bookmark, and the chapter's own number as Number: 391 for Chapter 391, 12.5 for an intermission after chapter 12 (Packing).

What a reader can't work out - a proper title, the author, a summary, genre, tags, and any other tag of the schema such as the publisher or the reading direction - comes from the comic's info block in mirror_metadata.json. The next append or pack writes it into the archive; a chapter archive takes everything but the title and address, which stay the chapter's own. Editing ComicInfo.xml inside a scraped comic's archive by hand doesn't last: the next append rewrites it from the metadata. Edit it instead with Edit ComicInfo in the comic's editor on The Web Page, which writes it into every archive the comic already has, or from the Reader, which can also change a whole folder or series at once, archives from elsewhere included. Number, Count, Notes, PageCount and Pages are always worked out and are not offered.

A comic scraped through a plugin keeps the ComicInfo.xml its downloader wrote, with the chapter it came in (or with the comic), as comicinfo in the metadata. It often knows more: a chapter's real number, the publisher, the language, the reading direction (Manga). Everything it says is written into the archive in the schema's order. Where two sources say the same thing, the info block wins, then the downloader, then what is worked out here. Notes carry both, the downloader's first. Its Pages, PageCount and Count are never used: the pages are described from the archive itself, and Count waits for the comic to be marked ended.

In a single archive the ComicInfo and metadata are the last two entries; an append rewrites them after the new pages, so every earlier byte stays put. A chapter archive works the same way: a new page goes on the end of its chapter's archive, and a changed ComicInfo on the end of each.

Older archives have none. This appends one to each, for a comic or a library. For a comic in chapters it writes only each chapter archive's ComicInfo (and renames any archive still named by its place, as packing does); pages scraped since are left for the next pack:

python chapters.py comicinfo "D:/Comics" --dry-run
python chapters.py comicinfo "D:/Comics"

Stand-in pages

Some pages are not pictures: a recording of an animated page, a linked video, or old Flash. They belong in the folder, but would read as gaps in an archive, so the archive gets a drawn PNG in their place saying what the page is and where. The original in the folder is never touched.

File in the folder	Stand-in says
.mp4, .m4v, .webm, .mov, .mkv	"This page is a video", and its filename
.swf	"This page was Flash", and its filename
.txt, .url or .webloc named like a page (0102.txt), 1 KB or less, holding a web address	"This page is a video", the note's title for it, and the address

The name and size rules keep a comic's readme from being taken for a page. The stand-in takes the original's name with .png, is identical in every kind of archive, and never breaks an address across lines.

Drawing one takes a moment, so it is drawn only when it goes into an archive, and once per run. Packing checks each chapter archive holds its stand-ins at the size they draw to, and remembers those sizes in the metadata (chapters.standins, keyed by what each stand-in says), so later packs don't draw them again. A stand-in whose note changes is drawn again, since what it says has changed.