ethanplusai/astro-conversion-kit/capture-site

capture-site

Save everything the live site serves, once, so every later step works from the capture.

The golden rule

The live site is read-only, and the capture is read-only afterwards. The scripts only make paced GET and HEAD requests; never submit a form, log in, or fetch by hand to fill a gap. Once the notes are written, nothing under capture/ is edited: it is the evidence every gate reads.

When to use

Step 2 of a conversion (convert-site runs it), or any time a live site has to be inventoried (a staging or new site is launch-audit's job). A capture that exited 3 resumes, as below.

Pacing

Every script sends a desktop Chrome user agent. The capture scripts (capture, media, assets, fingerprint) share a per-host limiter: by default request starts are 400 ms apart with up to 4 in flight (media: 6), so a host that answers quickly is read in minutes. The Chromium steps (shoot, css-coverage, styles) load one page at a time, 500 ms apart, and launch-audit and pre-launch keep their own pacing. The limiter adapts by itself:

• a 429 or 503 doubles that host's delay (up to 32 times) and drops it to **one request at a

time**; after every 30 s of clean answers it recovers one step (half the delay, one more in flight) until it is back to the defaults;

• an answer that takes more than 5 s halves the requests in flight on that host (not again while

halved), and it recovers the same way, one more in flight per 30 s of clean answers;

• a 403 or a network error counts towards the 3-in-a-row stop below, as before.

--polite is the old pace: one request at a time, 2500 ms apart (capture, media, assets). Use it from the start on shared hosting with a bot wall (SiteGround's /.well-known/sgcaptcha/, hosts that answer 429 or 503 to a few quick requests), or when a run stopped on blocks: resume with --polite. --delay <ms> and --concurrency <n> override the defaults or --polite. Keep the defaults or --polite against a live site (--delay 0 is for local fixtures only), and never change --ua to get past a block.

• capture fetches up to --concurrency queued URLs ahead and records them in queue order, so

the capture is the same as a one-at-a-time crawl. It tries each URL 3 times. A URL still blocked after that (403, 429, 503 or a network error) is recorded with its status and attempts in inventory.json (a sitemap or REST list in meta.json discoveryErrors) and the crawl moves on. It stops only after 3 different URLs in a row used up their tries with no success between them, state saved, those three queued again with fresh tries. It never requests wp-admin, wp-login, xmlrpc.php or ?s= searches.

• media tries each file 3 times: a persistent 429 or 503 stops it at once; a 403 or network

error is recorded as that file's status, and 3 such files in a row (in the order the answers arrive) stop it. Files already downloading when it stops are let finish. It saves media.json every 100 files, so a run that is killed resumes close to where it was.

• fingerprint does not retry: a blocked home page, or one that is a bot challenge, is exit 3. assets and shoot back off and

stop after 3 blocked responses in a row.

The command sequence

Run from the folder that will hold capture/ (the site repository's root). <skill-dir> is the absolute path of this skill's folder, in a copy of the kit where npm install has run (the scripts import its packages). Every script prints --help; run it rather than reading the source. Run each step as its own command, in this order:

1. Fingerprint, then read the platform reference for what it names (table below):

node <skill-dir>/scripts/fingerprint.mjs https://www.example.com

1. Capture. Discovery is the sitemaps (every Sitemap: line in robots.txt, plus the

platform's default index), the REST lists (WordPress, when the fingerprint finds REST; --no-rest skips), the link crawl, and --seed <path> or --seed-file <file> (one path or URL per line, # comments) for paths the crawl cannot reach (old URLs, an old .htaccess). After the crawl, Wayback paths it did not find go to wayback-unverified.txt (--no-wayback skips; a listing that fails is asked for once more 10 s later before meta.json wayback records failed). ?p= shortlinks are recorded unverified and never fetched unless you pass --verify-shortlinks. After the crawl it requests one path that cannot exist (/<32 hex>/) once, so the site's own 404 page is saved as pages/404.html (meta.json notFound; a site that answers 200 or redirects there is recorded with no page), and asks for /favicon.ico and /apple-touch-icon.png once each, redirects followed (meta.json icons: path to { status, file, contentType, bytes }); one that answers 200 is saved byte for byte as assets/root-favicon.ico or assets/root-apple-touch-icon.png (an HTML answer is not saved and has a note; one over 5 MB is status "too-large"), and assets.mjs lists it in assets/index.json as kind "root-icon" with its path, so the build ships it there. URLs are keyed with upper-case percent escapes (%c3%a9 and %C3%A9 are one URL), so a page linked with both spellings is captured once. A sitemap robots.txt names that answers 404 (or any non-200) is listed in sitemaps/index.json with its status and in meta.json discoveryErrors. A sitemap on another host (robots.txt naming a sister site's, or an index listing one; www and apex count as one host) is listed there as { url, host, skipped: "other host" } and never fetched, so none of its URLs enter the crawl. Two sitemap URLs with the same file name never overwrite each other: the second is saved with a suffix from its directory path (page-sitemap-es.xml). --exclude <prefix>[,<prefix>...] leaves parts of the host out (a WordPress multisite's sub-sites such as /president/, or a section the owner keeps): a URL whose path starts with one is never requested, whether a sitemap, the REST lists, a link or a redirect leads there. A sitemap under one is listed in sitemaps/index.json as { url, skipped: "excluded" }; a redirect into one is recorded with that target, unrequested; links to one stay in the saved HTML and are listed in links.json excluded, not internal; meta.json excluded names the prefixes and the baseline carries them. Prefixes and paths compare in one form, so /présidence/, /pr%c3%a9sidence/ and /pr%C3%A9sidence/ are the same prefix. A stylesheet or root icon that redirects under one is not followed there (the sheet is listed with status "excluded"). --resume keeps them; a prefix added on --resume removes what was already captured under it (its pages with their pages/ and text/ files, redirects from it, feeds and sitemaps under it, and stylesheets and scripts only those pages used) and prints what it removed. media.mjs and assets.mjs skip files under them too.

node <skill-dir>/scripts/capture.mjs https://www.example.com --out capture

Then the Wayback second pass, when wayback-unverified.txt is not empty: it requests each archived path once, so the ones that still redirect land in capture/redirects.json with source seed (the baseline's redirects pick them up) and the rest are recorded with their status in inventory.json. --seed, --seed-file and --include-query all apply on --resume, and the Wayback Machine is not asked again:

node <skill-dir>/scripts/capture.mjs https://www.example.com --out capture --resume --seed-file capture/wayback-unverified.txt

1. Media. Lists every media file in capture/media.json and saves it under

.conversion/media-cache/, the conversion's working directory, which stays out of git:

node <skill-dir>/scripts/media.mjs --capture capture --download .conversion/media-cache

It downloads six files at a time. By default (--sizes referenced) it fetches the originals and every size the site uses: one a page, stylesheet, srcset, og tag or sitemap points at, or whose file name appears anywhere in the captured pages, stylesheets or scripts (data-thumb, data-bg, JSON settings, slider scripts). A WordPress size only the REST media list names is listed with downloaded: false, reason: "unreferenced size" and not requested. It prints the split, e.g. 920 originals + referenced sizes (downloaded), 4,520 unreferenced sizes (listed only): repeat that line to the person. Use --sizes all when the owner says old emails or other sites may link to those sizes.

Hosts that resize by query string (WordPress VIP, Jetpack Photon: photo.jpg?w=1024, ?resize=300,200) serve every size at the same path. The original is the bare path, so that is what is requested and saved; each query string seen is listed under variants (not requested; --sizes all saves them as <path>%3F<query> under a sibling folder, .conversion/media-cache-variants/, so the media cache that is uploaded or copied into the repo holds bare paths only). When the bare path answers an error (4xx or 5xx; a redirect is recorded with its location, and a too-large file is asked again once the cap is raised), a variant is saved in its place and the entry's downloadedFrom is the URL it came from. A cache-busting query (no w, h, resize, crop, fit, quality, format or s parameter, e.g. ?ver=3) is the same file, so it is preferred and nothing else is marked; otherwise the widest variant is saved and the entry also says resized: true. The run prints a line per file and a count (2 originals unavailable (resized: true) ...), baseline/media.json keeps both fields, and the media gate repeats the count as N resized (original unavailable on live) without failing, since it compares against the bytes actually saved. Repeat that count to the person and list those files in the plan and runbook as originals the live site no longer serves.

1. Templates. Clusters the pages that answered 200 into capture/templates.json. On

WordPress, pages group by template body class first (home, page, single-post, archive, template classes) and a page whose markup is clearly different (Jaccard below 0.5) splits off as page-2; other platforms merge by markup at Jaccard 0.6, as template-<n>:

node <skill-dir>/scripts/templates.mjs --capture capture

1. Assets. Logo, favicon, apple-touch-icon, the <link rel=manifest> web manifest and every

@font-face font into capture/assets/, plus the root /favicon.ico and /apple-touch-icon.png (root-favicon.*, root-apple-touch-icon.*) when meta.json icons recorded them at 200:

node <skill-dir>/scripts/assets.mjs --capture capture

The logo is looked for on the home page inside the <header> first, then inside any other element whose class or id names a logo (.logo, .site-logo, #logo-wrap). In each: an <img> whose src, alt, class or id has the word "logo" (not "logout"), then an inline <svg> with "logo" in its class, id, aria-label or <title> (saved as logo.svg, url: null in assets/index.json; its note says so when it only <use>s a symbol defined elsewhere on the page, which renders empty on its own), then the first <img> in a link to the home page, then the first <img>. Check assets/logo.* by eye: an account avatar or a sponsor mark can still win on an unusual header. The favicon and apple-touch-icon are saved as served from their <link> href, query string included, so one that points at a resize (?w=32, ?w=180) of an uploads file differs in bytes from that file's entry in media.json: keep the saved icon, not the original.