The XPaths that find a page's comic image, next link, first-page link and age gate's button, and how to add your own (for exit code 3, a redesign, several pages on one address, or a site that asks your age).

Each list is tried in order until one matches; the winner is remembered per comic and tried first next time. The first list is only used by a walk to get back to page 1, and the gate list only on a page opened by its address (below). The scripts ship a short generic set in comiclib/elements.py, mainly as examples; most libraries add a few of their own.

1. See what matches. On The Web Page, open Element paths, paste a mid-comic page address (not the newest) and press Check this page. If image and next both match, you are done. (The first-page match only matters for a comic walked from a later page.)

   

2. Take a suggestion. With no match it suggests paths from the page's images and links, each with add it; the first is usually right, and lands at the top of its list noted with the site's name.

   

3. Or write one. In your browser, Inspect the comic and look for an id or class that names it. Test in the console with $x('//img[@id="strip"]').

4. Save, then run the comic or check again.

Without the web page, python mirror_base.py --check "https://example.com/comic/some-page" does the same, saving nothing.

• Image paths end on the <img>, whose src is saved: //*[@id="comic"]//img, not //*[@id="comic"].
• Use ids and classes, not positions: //div[@id="strip"]/img survives a redesign; Copy XPath's /html/body/div[3]/div[1]/img does not.
• Keep next paths to <a>: a <link rel="next"> matches too but goes nowhere. Next links to another site are never followed.
• Order matters: the first match wins, so put comic-specific paths above general ones (drag the handle or use the arrows).
• Note what it is for, usually the comic's name; it is the only record of why it is there.

--check runs JavaScript; a scrape does not (pages load faster). So --check compares what the browser found with what the server sent, and says when JavaScript is the difference:

image: 1 of the known paths match; a scrape would use //*[@id="content"]/img
  that image is not in the page the server sends, only in the one javascript builds, so this comic
  needs javascript as well as the path: -ej, or "javascript": true in its settings.

A run that finds no image says the same from its side.

The Element paths dialog's Export and Import… pass paths between two libraries without touching anyone's config folder by hand.

• Export saves element-paths-<date>.json with the paths you added, all three lists. Shipped paths are left out, since the other copy has them already. To pick what goes, tick rows' export boxes (shipped ones too), and the button becomes Export ticked, saving only those. Each path keeps its note and whether it is on.
• Import… reads an exported file, or anyone's element_paths.json. Its paths go to the top of their lists, in the file's order, so they are tried before the general ones, and are marked until saved. A path you already have is skipped and stays exactly as yours is. One that is not an xpath is skipped too, and the dialog says how many of each.
• Nothing is written until Save, so you can reorder, turn off or remove what came in first.

The web page writes config/element_paths.json; editing by hand works too:

{
  "image": [{ "xpath": "//img[@class=\"strip-art\"]", "note": "Odd Comic", "enabled": true }],
  "next":  [{ "xpath": "//*[@class=\"onwards\"]", "note": "Odd Comic", "enabled": true }],
  "first": [{ "xpath": "//a[@class=\"to-start\"]", "note": "Odd Comic", "enabled": true }],
  "gate":  [{ "xpath": "//button[@id=\"enter\"]", "note": "Odd Comic", "enabled": true }]
}

• The file sets order and what is off. Shipped paths it omits are tried after it, so new shipped paths still arrive; they can be turned off but not deleted, so an update cannot quietly bring one back.
• An unreadable file is reported and ignored.
• It is looked for in the config folder, then the starting folder, then beside the script; MIRROR_ELEMENTS names it outright.
• Site-specific paths belong here, not in the code.

Some sites show an "are you 18?" page first, and only show the comic once its button is pressed. The button sets a cookie, and the site checks it on every page and often every picture. Without it, a scrape sees only the gate: exit 3, or worse, a picture on the gate (a warning, a banner) taken for the comic.

The sure way is the cookie itself. Pass the gate once in your own browser, open its developer tools (Application → Cookies in Chrome, Storage in Firefox) and find what it set: AGE_CONFIRMED=yes, consent=given and the like. Then type it into the comic's Cookies box on The Web Page, or give it on the command line with --cookie NAME=VALUE. Several are put apart with ; in the box, or by repeating --cookie. It is kept in the comic's settings (Settings and Metadata) and goes with everything that reads the site:

• the browser, before it loads the first page, so the gate never appears
• every picture fetched, sent to the comic's site and its neighbours (static.example.com beside www.example.com) and nowhere else
• walks, archive pages, refetch and insert (see Chapters)

A gate's cookie usually lasts a year in a browser, but the comic's copy never expires, since it is sent fresh every run. It is for a gate's answer, not a login: it is written in the comic's metadata and shows in its run record.

Or a gate path. The gate list holds the button on a gate. When a page opened by its address (a run's first page, or the one a fresh browser picks up from) matches one, it is pressed before anything else is looked for, so nothing on the gate can become the comic's image path. Then the page wanted is opened again, since many gates send you to the front page. The cookie it set goes with the pictures for the rest of that run. The shipped paths only know the commonest wordings; a gate that says something else needs its own.

https://example.com/age-check/?next=... is the site's age gate; pressing //button[@id="enter"] to pass it.

A gate whose button sets its cookie with a script does nothing in a scrape, which runs without JavaScript. The run says so, and points to --cookie. Check this page runs JavaScript, so it can pass a gate a scrape cannot; it says when it pressed one.

To check a page behind a gate with no gate path, put the gate's cookie in the box beside the address (consent=given, several put apart with ;, as in the comic's Cookies box). Without it the check sees only the gate, and suggests paths for the gate's own pictures and buttons.

Some comics switch to serving a scroll of pages, or six, per address; saving only the image would silently lose most of them. So every image the path matches is saved in order, numbered on from the last:

Unnumbered matches (back/next buttons, an RSS icon) are dropped, and the run says so; if none are numbered, all are kept. --check reports it up front (that path finds 14 images here, 14 of them pages), and a run logs the first time it happens:

https://example.com/comic/270 holds 6 pages of the comic where every address so far held one, so this
comic has started putting several pages on one address. Each is saved in turn, numbered on from the last.

state records pages_per_url and multi_page_from, and the index gets a line per page, so chapters still align. --no-multi-page takes one page per address and is remembered in the comic's settings. Use a path scoped to the container (//*[@id="pages"]/img); --check suggests one for images with no id, class or alt.

A comic published a week to an address often fills it a day at a time, with the same "coming soon" picture in every slot not out yet. Saved, that picture is a page: numbered after the last real one, sorted before the first (a name like 000.jpg reads as page 0), and in the place the real page will need. Alignment then can't place it, so the comic can't be cut into chapters.

Name that picture in the comic's Edit, under Advanced settings, as Not out yet: its address from the browser's Copy image address, or its path from the first /. On the command line it is --placeholder ADDRESS, repeatable. It is never saved. A run saves the pages before it, stops as caught up, and the next run starts again at that address, so each page is numbered where it stands as it comes out. A walk for chapters stops there too. Kept in the comic's settings as placeholders (Settings and Metadata).