secutils-dev/secutils/api

Secutils.dev: Web Scraping (API Trackers)

An API tracker issues an HTTP request to a URL on a cron schedule, runs an optional configurator script to mutate the request before it goes out (add headers, sign payloads, expand templated bodies), and an optional extractor script to transform the response into the value that gets persisted as the latest revision. Both scripts run inside the Retrack Deno sandbox documented in secutils-deno-runtime.

Use API trackers whenever a browser is unnecessary: JSON APIs, status endpoints, GraphQL queries, RSS feeds, plain-text endpoints. For pages that need full DOM rendering or interaction, use the page tracker documented in secutils-web-scraping-page.

Guide: <https://secutils.dev/docs/guides/web_scraping/api>. Full reference: <https://secutils.dev/api-docs/openapi.json>.

Endpoints (tag: web_scraping)

Authenticate with the Kratos session cookie or Authorization: Bearer su_ak_<token> (see secutils-api-keys).

Create-tracker payload

POST /api/web_scraping/api_trackers accepts ApiTrackerCreateParams:

{
  "name": "App state",
  "enabled": true,
  "config": {
    "revisions": 10,
    "timeout": 10000,
    "job": { "schedule": "0 0 * * * *" }
  },
  "target": {
    "requests": [
      {
        "url": "https://api.example.com/state",
        "method": "POST",
        "headers": { "content-type": "application/json" },
        "body": "{\"key\":\"value\"}",
        "acceptInvalidCertificates": false
      }
    ],
    "configurator": null,
    "extractor": null
  },
  "notifications": true,
  "secrets": "none",
  "tagIds": []
}

Field notes:

• target.requests is an array; the request at index 0 is the primary

request. When the array has more than one entry, all requests are issued in order and the extractor sees every response in context.responses.

• target.requests[i].body is sent verbatim as the HTTP body. JSON

bodies must be pre-stringified.

• target.requests[i].acceptInvalidCertificates disables TLS

verification for self-signed development endpoints; do not enable in production.

• target.configurator is an IIFE (() => { ... })() that returns a

mutated requests array. Use it to inject Authorization headers from context.params.secrets.*, sign payloads, or expand templated URLs from secrets.

• target.extractor is an IIFE that receives

context.responses: Array<{ status, headers, body: number[] }> and returns { body: Uint8Array } (or any of the auto-converted forms; see secutils-deno-runtime). When omitted, the raw body of the last response is stored.

• secrets follows the same none / all / { "selected": [...] }

union used by responders and page trackers (see secutils-secrets).

Configurator and extractor contracts

Configurator script

(() => {
  const token = context.params?.secrets?.AUTH_TOKEN ?? '';
  return {
    requests: context.requests.map(r => ({
      ...r,
      headers: { ...r.headers, authorization: `Bearer ${token}` }
    }))
  };
})();

Extractor script

(() => {
  const last = context.responses?.[context.responses.length - 1];
  if (!last) return { body: Deno.core.encode('no response') };
  const json = JSON.parse(Deno.core.decode(new Uint8Array(last.body)));
  return { body: Deno.core.encode(JSON.stringify({ status: json.status })) };
})();

Debug / test without saving

POST /api/web_scraping/api_trackers/_debug accepts the same target, config, and secrets fields as the create endpoint and returns:

{
  "configurator": { "result": { ... }, "params": { ... }, "error": null },
  "requests":     [ { "request": { ... }, "response": { ... } } ],
  "extractor":    { "result": { ... }, "params": { ... }, "error": null },
  "result":       { "body": "..." },
  "elapsedMs":    123
}

_test is the lighter variant: it issues the HTTP request without running the configurator or extractor and returns the raw response. Agents should prefer _debug when validating an end-to-end pipeline.

Example flow (curl)

# Create a JSON tracker
TRACKER=$(curl -sX POST https://secutils.dev/api/web_scraping/api_trackers \
  -H "Authorization: Bearer $SECUTILS_API_KEY" \
  -H "content-type: application/json" \
  -d @api-tracker.json)
ID=$(echo "$TRACKER" | jq -r '.id')
 
# Newest revision first
curl -sX POST "https://secutils.dev/api/web_scraping/api_trackers/$ID/_history" \
  -H "Authorization: Bearer $SECUTILS_API_KEY" \
  -H "content-type: application/json" \
  -d '{"refresh":true}' | jq '.[0]'

Caveats

• The Deno sandbox does not expose fetch; requests are made by the

Retrack runtime based on target.requests, not from inside the scripts. Configurators can only declare what to send; extractors can only read what came back.

• Compressed responses (gzip, deflate, brotli) are decompressed before

the extractor sees them, so JSON.parse(Deno.core.decode(...)) works regardless of Content-Encoding.

• Free tier minimum schedule interval is 1 hour; paid tiers allow

shorter intervals.

• Storage is capped at 1 MB per revision; larger responses are

truncated by the extractor (or by the default extractor when none is provided).

• A tracker with secrets: "none" running a configurator that tries

to read context.params.secrets.X will receive undefined, not throw. Always coalesce with ?? '' to avoid producing malformed request payloads.

See also

• Human-readable guide: <https://secutils.dev/docs/guides/web_scraping/api>
• Related skill: secutils-web-scraping-page (full headless browser)
• Sandbox API reference: secutils-deno-runtime
• OpenAPI: <https://secutils.dev/api-docs/openapi.json>