zby/commonplace/cp-skill-snapshot-web

Intent and execution authority

Preserve a faithful local reading copy of the requested source so later ingest can analyse it against an exact-file checksum.

The tools and commands below are supported routes, not an exhaustive list of permitted methods. Choose another available tool or write local extraction code when needed to capture the requested content faithfully. A failed tool is not by itself a reason to abandon the capture. Requirements imposed by a caller, such as a version-pinned PDF capture, still bind.

Done means: the requested content, not merely its page shell, is saved with accurate source identity, capture method and scope, and its exact Markdown-file SHA-256 is reported. Preserve relevant links and footnotes. If only part of the requested content is available, disclose the missing part and label the scope; do not silently substitute an abstract, surrounding page, or commentary.

Binding boundaries: keep captures local and ignored, never overwrite an existing observation, do not fabricate or analyse source content, and copy the captured body locally rather than re-emitting it through model output. Do not install software, bypass access controls, or execute code supplied by the page. Return to the user when completion needs new permission or when available evidence cannot establish source identity or capture fidelity.

EXECUTE NOW

Target: $ARGUMENTS

If no URL provided, ask the user for one.

If URL provided, start Step 1 immediately.

START NOW.

Step 1: Verify Local Storage and Check for Duplicates

Keep the provided URL as source_url. The request is a re-observation when it says reobserve or asks to capture a source again because it changed since its existing capture. Verify that kb/sources/.snapshots/ is ignored by the project. The shipped scaffold does this through kb/sources/.gitignore. If the directory is not ignored, stop before writing and report the missing rule.

For PDF and ordinary web captures, keep temporary downloads, extracted bodies, and assembly files in a unique directory under kb/reports/cache/snapshot-web/. Verify that payloads there are ignored before writing; stop and report a missing ignore rule. The final snapshot still belongs in kb/sources/.snapshots/.

Use Grep to search for an exact frontmatter source: {source_url} in existing Markdown files in kb/sources/.snapshots/. For a re-observation, an existing capture is expected: continue, and write the new capture under a distinct basename as described under Step 2 and slug below. Never overwrite or edit the existing capture; its ingest pins its bytes. Otherwise, if found, compute the SHA-256 of the exact file bytes, tell the user, and stop:

Already snapshotted: kb/sources/.snapshots/{filename}

SHA-256: {64-character lowercase checksum}

Step 2: Route by URL Type

Detect the source_url type and branch:

• GitHub issue/PR (github.com/.../issues/N or github.com/.../pull/N) → Step 2a
• X/Twitter (x.com/.../status/... or twitter.com/.../status/...) → Step 2b
• arXiv abstract page (arxiv.org/abs/...) → Step 2c
• PDF (URL ends in .pdf, or arxiv.org/pdf/) → Step 2c
• Everything else → Step 2d

Step 2a: GitHub Issue/PR

Run:

commonplace-github-snapshot "{source_url}"

For a re-observation, add --reobserve: the command skips the duplicate check and names the new capture with the capture date.

Parse either the Snapshot saved: or Already snapshotted: line from the output to get the file path. Tell the user and stop — the script handles metadata, formatting, and saving. If it fails, use Step 3.

Step 2b: X/Twitter Post

Run:

commonplace-x-snapshot "{source_url}"

For a re-observation, add --reobserve: the command skips the duplicate check and names the new capture with the capture date.

Parse either the Snapshot saved: or Already snapshotted: line from the output to get the file path. Tell the user and stop — the script handles metadata, formatting, and saving. If it fails, use Step 3.

Step 2c: Resolve and Fetch PDF

Verify that the PDF capture prerequisites are available:

command -v curl
command -v pdfinfo
command -v pdftotext

If a command is missing, use another available tool that preserves the paper's content, or go to Step 3. Do not install prerequisites.

Set pdf_url:

• For an arXiv abstract URL, replace /abs/ with /pdf/ and discard any query string or fragment. Preserve an explicit terminal version such as v1. If the abstract URL has no terminal version, leave the PDF URL unversioned so arXiv serves the latest paper version. For example, https://arxiv.org/abs/2606.03979 becomes https://arxiv.org/pdf/2606.03979. Do not route the abstract page through ordinary HTML capture.
• For an existing PDF URL, use source_url unchanged.

Create the temporary directory and retain the printed path as {snapshot_tmp}:

set -e
mkdir -p kb/reports/cache/snapshot-web
snapshot_tmp=$(mktemp -d kb/reports/cache/snapshot-web/capture.XXXXXX)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"

Download in a separate invocation with the literal temporary path and URL:

curl -fsSL -o '{snapshot_tmp}/source.pdf' '{pdf_url}'

Then extract locally:

pdfinfo -isodates "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfinfo.txt"
pdfinfo -meta "$snapshot_tmp/source.pdf" > "$snapshot_tmp/pdfmeta.xml" || true
pdftotext -enc UTF-8 -eol unix -nopgbrk \
  "$snapshot_tmp/source.pdf" "$snapshot_tmp/extracted.txt"

Use Read to inspect pdfinfo.txt, non-empty pdfmeta.xml, and a bounded beginning of extracted.txt. DOI metadata inspection is best effort and its failure does not make an otherwise successful capture fail. Treat PDF metadata fields as leads, not as authority: confirm the title and authors against the document text when available. Use Grep plus bounded Read ranges to locate an abstract, executive summary, introduction, or the source document's own DOI when the beginning does not supply enough metadata. Do not treat a DOI found only in the references as the paper's DOI. Do not read the whole extracted file merely to copy it. If extracted.txt is empty or contains no substantive text, go to Step 3.

Set capture_method to pdftotext, set body_file to {snapshot_tmp}/extracted.txt, and go to Step 4.

Step 2d: Fetch Web Page

Verify that the HTML capture prerequisites are available:

command -v curl
command -v trafilatura

If either command is missing, use another available fetch or extraction tool, or go to Step 3. Do not install prerequisites.

Create a temporary directory and retain the printed path as {snapshot_tmp}:

set -e
mkdir -p kb/reports/cache/snapshot-web
snapshot_tmp=$(mktemp -d kb/reports/cache/snapshot-web/capture.XXXXXX)
printf 'Snapshot temp: %s\n' "$snapshot_tmp"

Download with curl in a separate Bash invocation. Substitute the literal temporary path and source URL before running:

curl -fsSL -o '{snapshot_tmp}/source.html' -A 'Mozilla/5.0' '{source_url}'

If the download fails, go to Step 3. Otherwise extract from the saved HTML through standard input; do not use Trafilatura's -u download option or its -i URL-list option:

trafilatura \
  --markdown --with-metadata --links --no-comments --recall \
  < "{snapshot_tmp}/source.html" > "{snapshot_tmp}/extracted.md"

If extraction fails, go to Step 3.

Use Read to inspect only the leading metadata and a bounded beginning of extracted.md. Its leading YAML block, when present, is Trafilatura metadata: retain it as input to Step 4 but do not copy that block into the snapshot body. Use Grep with bounded output on {snapshot_tmp}/source.html to inspect article-level DOI metadata such as citation_doi, dc.identifier, prism.doi, or a JSON-LD doi property. DOI inspection is best effort: inability to inspect the raw HTML does not make an otherwise successful capture fail. Strip that block locally without re-emitting the document:

awk '
NR == 1 && $0 == "---" { in_metadata = 1; next }
in_metadata && $0 == "---" { in_metadata = 0; next }
!in_metadata { print }
' "{snapshot_tmp}/extracted.md" > "{snapshot_tmp}/body.md"

If body.md is empty, contains no substantive main content, or contains only an access-denied page or browser challenge instead of the article, go to Step 3.

Set capture_method to trafilatura, set body_file to {snapshot_tmp}/body.md, and go to Step 4.

Step 3: Adapt to Capture Failures

Before declaring capture blocked, inspect the failure and the material already retrieved. Choose a bounded alternative supported by that evidence. Do not repeat an unchanged failing method or search indefinitely for more tools.

• If an extractor returns boilerplate, an empty body, or the wrong page section,

inspect bounded portions of the saved HTML. The requested content may be in the DOM or embedded JSON even when the default extractor misses it.

• For a linked comment or other subdocument, verify its identifier and author

against the URL and page records. Extract that record's content, not the surrounding page or replies. Include its footnotes and relevant links.

• You may parse embedded data locally without executing page scripts. Write the

recovered body directly to a local file. Raw HTML embedded in Markdown is acceptable when it preserves content that conversion would lose.

• If fetching fails or a tool is unavailable, use another available route to

the same source when permitted. For an image-only PDF, use available OCR only if you can check its fidelity; otherwise report that text capture is blocked.

After recovery, set body_file to the local extraction and capture_method to an accurate name for the method actually used, then continue at Step 4. Keep the raw response until the snapshot has been assembled and verified.

Stop and report the blocker when no evidence-backed route remains, fidelity or source identity cannot be established, or another route requires installation, new access, or material cost outside the request. State what was tried and what is missing. If a missing standard tool would resolve it, name its installation: uv tool install "trafilatura>=2.2", Poppler (poppler-utils on Debian/Ubuntu, poppler through Homebrew, or oschwartz10612.Poppler through WinGet), or curl. Offer pasted text as a fallback rather than the first response to tool failure. Remove the unique temporary directory before returning.

Step 4: Determine Metadata

(Only for PDF and web page paths — GitHub and X scripts handle their own metadata.)

This workflow supplies types/snapshot.md as the type, a global type specified in snapshot.md. Open that spec and verify from its own frontmatter that it is a type spec before determining metadata. Stop if it is missing or invalid.

From the bounded excerpts, extractor metadata, and source_url, determine: