secutils-dev/secutils/page

Secutils.dev: Web Scraping (Page Trackers)

A page tracker schedules a headless Playwright (Chromium) navigation to a URL, runs a user-defined execute(page, context) function inside the Retrack scraper sandbox, stores the returned value as the latest revision, and optionally notifies the user when the value differs from the previous revision. Trackers can also extract structured script and stylesheet resource descriptors (URL, content hash, size) for synthetic monitoring of deployed web apps.

Unlike API trackers (see secutils-web-scraping-api), page trackers launch a real browser, so they handle SPA hydration, lazy-loaded assets, and DOM interactions. The trade-off is higher latency per run and a single 1 MB cap on the extracted body. Pages behind a CAPTCHA or web application firewall are not supported.

Guide: <https://secutils.dev/docs/guides/web_scraping/page>. Full reference: <https://secutils.dev/api-docs/openapi.json>.

Endpoints (tag: web_scraping)

Authenticate with the Kratos session cookie or Authorization: Bearer su_ak_<token> (see secutils-api-keys).

Create-tracker payload

POST /api/web_scraping/page_trackers accepts PageTrackerCreateParams:

{
  "name": "Hacker News - Top Post",
  "enabled": true,
  "config": {
    "revisions": 5,
    "timeout": 30000,
    "job": { "schedule": "0 */15 * * * *" }
  },
  "target": {
    "extractor": "export async function execute(page) { await page.goto('https://news.ycombinator.com/'); const a = page.locator('.athing .titleline a').first(); return `[${await a.textContent()}](${await a.getAttribute('href')})`; }",
    "userAgent": null,
    "ignoreHttpsErrors": false
  },
  "notifications": true,
  "secrets": "none",
  "tagIds": []
}

Field notes:

• config.revisions is the rolling history depth (1..100). Older

revisions are dropped on overflow.

• config.timeout is per-run wall clock in ms (default 30000, max

120000).

• config.job.schedule is a 6-field cron expression (seconds, minutes,

hours, day-of-month, month, day-of-week). Omit job to make the tracker on-demand only.

• target.extractor must export async function execute(page, context)

and return a serialisable value. The returned value is JSON-stringified (or kept as bytes if the script returns Uint8Array) and capped at 1 MB.

• target.userAgent overrides the default Chromium UA when set.
• secrets follows the same none / all / { "selected": [...] }

union used elsewhere (see secutils-secrets). Secrets are exposed via the second argument: execute(page, context) with context.params.secrets.MY_KEY.

Debug-without-saving

POST /api/web_scraping/page_trackers/_debug accepts the same target and config fields and returns the full pipeline trace: navigation timings, console messages, the extractor's return value, and any thrown error. Use it from agents to validate an extractor against a URL before persisting the tracker.

Resource extraction shortcut

The Retrack scraper ships a helper that returns Array<{ type: 'script' | 'stylesheet', url?: string, content: { size: number, digest: string } }> for the visited page. The human guide includes a full example; in short the extractor returns the result of Deno.core.ops.op_extract_resources() to get a stable list suitable for change detection.

Example flow (curl)

TRACKER=$(curl -sX POST https://secutils.dev/api/web_scraping/page_trackers \
  -H "Authorization: Bearer $SECUTILS_API_KEY" \
  -H "content-type: application/json" \
  -d @page-tracker.json)
ID=$(echo "$TRACKER" | jq -r '.id')
 
# Read the latest revision (sorted newest first)
curl -sX POST "https://secutils.dev/api/web_scraping/page_trackers/$ID/_history" \
  -H "Authorization: Bearer $SECUTILS_API_KEY" \
  -H "content-type: application/json" \
  -d '{"refresh":false}' | jq '.[0]'

Caveats

• Pages protected by CAPTCHA, web-application firewalls, or strict bot

detection (navigator.webdriver checks) often fail. Use an API tracker against an internal endpoint when possible.

• The extractor cannot make outbound HTTP requests (fetch,

XMLHttpRequest); use page.request for HTTP, or proxy through a Secutils responder.

• config.job.schedule is on a best-effort scheduler shared across

users. Free tier minimum interval is 1 hour; paid tiers allow shorter intervals.

• Notifications are sent through the email channel configured on the

account; if the user has not verified an email address, notification delivery silently fails.

• The recording-import shortcuts (`Right-click → Import: Playwright

recording / Chrome DevTools recording) are UI-only; the API expects a fully formed execute() function in target.extractor`.

See also

• Human-readable guide: <https://secutils.dev/docs/guides/web_scraping/page>
• Related skill: secutils-web-scraping-api (HTTP-only tracker; lighter

and faster when a browser is unnecessary)

• Sandbox API reference: secutils-deno-runtime
• OpenAPI: <https://secutils.dev/api-docs/openapi.json>