version: "1.0.0" name: sdl-log-parser author: Prithvi Moses <prithvi.moses@sentinelone.com> description: Use whenever the user wants to author, edit, debug, validate, or explain a SentinelOne Singularity Data Lake (SDL) log parser, the augmented-JSON files at /logParsers/ that extract fields from raw log text before ingestion. Trigger on "SDL parser", "Skylight parser", "log parser", "parser editor", "write a parser", "test parser", or any pasted raw log the user wants turned into structured fields. Also trigger on parser-DSL keywords like formats:, patterns:, lineGroupers:, rewrites:, discardAttributes:, aliasTo:, {parse=...}, {regex=...}. Especially trigger when the user pastes a raw log (CEF, syslog, JSON, key=value, multi-line, CSV) and asks to extract fields, normalize timestamps, or drop noise. If the project is SDL/Singularity/Scalyr and the user says "parse this log", use this skill. Always validates end-to-end via putFile → HEC ingest (parser applied) → query. NOT for PowerQuery (use powerquery), NOT for plain ingest without a parser (use sdl-api).

SentinelOne SDL Log Parser Authoring

<!-- CONFIG-FILE-ADDRESSING v1 -->

SDL config files: address by `udoId`, and do not trust a REST listing.

REST listFiles / getFile cannot see udoId-addressed /dashboards/ files, so getFile

returns 404 on a dashboard the console is displaying and the listing under-reports (measured on

one tenant: REST 8, GraphQL 17, console 48 files). **If a listing disagrees with what the UI

shows, the listing is wrong until proven otherwise**, change read path before concluding the

object is missing or the token lacks scope. Use the GraphQL configFiles / configFile surface.

A name-addressed addConfigFile to /dashboards/ creates a duplicate instead of updating;

address dashboards by udoId with expectedVersion. content is HJSON, not JSON. S1-Scope

changes which files exist as far as the caller can tell.

Full detail: `sdl-api/references/config-file-graphql.md`

This skill turns raw log samples into deployed, validated SDL parser definitions. A parser is an augmented-JSON file at /logParsers/<name> on the SDL tenant that extracts fields from each ingested line. The parser editor and the Test Parser button in the console run the parser client-side in JavaScript; this skill mirrors that workflow programmatically and finishes by ingesting a sample through the deployed parser to confirm the actual ingest path works.

Deploy and validate through the `s1-secops-mcp` MCP tools. sdl_get_file / sdl_put_file read and deploy the parser, powerquery_run validates on the live stream, and hec_ingest runs the synthetic-ingest fallback. The server runs on the user's machine, so it reaches *.sentinelone.net where the Cowork sandbox cannot, and it reads credentials from environment variables or the OS keychain. If the tools are missing, point the user to the s1-secops-mcp README (s1-secops-mcp/README.md in the s1-secops-skills source repo, mcp/s1-secops-mcp/README.md in ai-siem; not shipped inside the plugin).

Gate 0: can an SDL parser run on this source at all? (check this first)

The SDL parser sees exactly two attributes on an event: message and parser. Every other attribute on the event is invisible to it. The value of the parser attribute names which parser to apply, and that parser is applied to the content of message. That one fact decides whether writing a parser is the right move, so settle it before you invest in authoring.

An SDL parser cannot reach fields that live outside `message`. If the fields you want to transform are already promoted to top level on the event, an SDL parser is the wrong tool however it is written: it has no way to read them. That work belongs in DPM.

Determine which case a source is in by inspecting a sample event for the presence of message and parser. The PowerQuery probes for that, plus how to set the parser attribute from DPM, are in references/parser-eligibility.md.

Two hard rules (every parser, every time)

These two rules apply to every parser you write or edit, regardless of source, regardless of whether you're starting from scratch or copying from the catalog. Violating either produces a non-conformant parser even if extraction works.

Rule 1: The 4 mandatory attributes are always present.

attributes: {
  "dataSource.category": "security",   // hardcoded constant, never any other value
  "dataSource.name":     "Corelight",  // specific product/service name
  "dataSource.vendor":   "Corelight",  // parent vendor / company name
  "metadata.version":    "1.0.0"       // semver; bump on substantive parser changes
}

dataSource.category is fixed at "security". Not application, not network, not system, not audit. The catalog has parsers that violate this; you do not.

metadata.version is mandatory and follows semver (MAJOR.MINOR.PATCH, with optional pre-release suffix like -rc1 or -beta1). Increment it on every build. Patch bump for fixes, minor for additive changes, major for breaking schema changes. It can also be set inside mappings via a constant op when the source itself carries a vendor schema version you'd rather propagate per-event:

{ constant: { field: "metadata.version", value: "<your-current-version>" } }

A constant in mappings overrides the parser-root attributes: value when it fires, which is useful when one parser handles multiple vendor schema versions and you want each event tagged with the right one.

Rule 2: Every OCSF field name comes from `references/ocsf-schema-documentation.md`.

Before emitting any OCSF dotted path (src_endpoint.ip, actor.user.name, metadata.product.vendor_name, etc.), grep references/ocsf-schema-documentation.md for the exact string. Do not invent field names. Do not copy from a catalog parser without verifying. Do not rely on memory. The reference is the single source of truth for the ~25,759 documented OCSF field names across the 7 categories.

The rest of this skill assumes both rules are upheld. If either is unclear for a given parser, stop and resolve before continuing.

Critical pitfalls (read before authoring)

These are the DSL traps that cost the most iteration cycles when authoring parsers on the current S1 tenant. They are repeated in the long gotcha list below with more context, but if you're about to write a new parser, internalize these first. Items 1-9 are tenant-validated as of 2026-05-26; items 10-12 as of 2026-06-22.

1. Key=value catch-all REQUIRES leading `.*`. The canonical form is ".*$_=identifier$=$_=quoteOrSpace$" with repeat: true. Without the .* the format anchors at position 0 of the message and fires zero times on any line starting with a syslog priority <NNN>, a CEF header, an XML envelope, or a timestamp, i.e. virtually every real log source. The leading .* lets the engine scan forward to the first key=value pair and then repeat: true walks the rest. This is the same pattern the built-in keyValue parser uses. Also: `{parse=keyValue}` is NOT a valid sub-parser directive; built-in parsers like keyValue are only usable via root-level aliasTo:.
2. v1 mappings are FIRST-MATCH-WINS, which means renames must be duplicated into every per-subtype block alongside the constants. A catch-all predicate: "true" rename block placed at the end NEVER fires for events that matched an earlier per-subtype block, only the truly-unmatched events get those renames. If you have per-subtype constant blocks for OCSF class/activity fan-out (e.g. subtype=='forward' → activity_id=6), you must put the full rename list inside EACH of those blocks. Verbose but correct.
3. v0 constants take `field`, not `output`. A v0 constants: entry keyed output (the key v0 renames: and copies: use) is rejected by putFile ("Expected a field", surfaced by the MCP as "Content is invalid json"). Keyed field ({ field: "class_uid", value: 4001, type: "int" }) it deploys and applies, which is the shape every marketplace v0 parser uses. Live-validated 2026-10-13; this corrects an earlier note that all v0 constants: were rejected. For new parsers prefer v1 ({ constant: { field, value } }) anyway.
4. Same-name v1 rename silently NULLS the field (a same-name v0 renames entry keeps the value, live-validated 2026-10-13). Writing { rename: { from: "action", to: "action" } } to "keep" a captured raw field under its own name actually deletes the value. The KV catch-all already extracts raw vendor field names directly into the event, to "keep" one, OMIT the rename. Only write a rename when the target name is genuinely different from the source.
5. Unquoted numerics from `quoteOrSpace` are auto-typed as NUMBER columns. A FortiGate srcport=44321 sentbyte=8421 line going through the KV catch-all produces srcport: 44321 (number), sentbyte: 8421 (number), queryable with avg() / sum() / > N without number() wrapping. Quoted values stay as strings even if they look numeric (level="6" → string). Caveat: SDL column type-locking still applies, so wrap numerics with number() in dashboard panels and STAR rule bodies if the column has ever been written as string in this tenant's history.

Two related queryability rules (not DSL traps, but they matter for any consumer of the parser):

1. The raw `message` field is preserved on every event regardless of parser activity. SDL never lets a parser overwrite message; the raw event body always survives. So parse "field=$x$" from message queries continue to work alongside the typed columns the parser produces. This is the backward-compat escape hatch during a parser rollout.
2. When two ingest paths produce OCSF for the same source, the schema can drift. A common case: upstream collector (Observo etc.) flattens type=traffic, subtype=forward to just event.type='forward' while the downstream parser keeps both fields populated. Consumers (dashboards, STAR rules, hunts) should write subtype filters as (event.subtype='forward' OR event.type='forward') until the rollout fully converges.
3. Trailing `.*` at the END of a format string suffers the leading-`.*`-strip too. A field marker like $process.cmd_line{regex=.*}$ placed at the end of a format string captures the EMPTY STRING. The SDL stripping rule applies whether .* starts the regex or IS the entire regex. Use [\\s\\S]+ (one-or-more) or [\\s\\S]* (zero-or-more) for "everything that remains on the line" semantics; those forms don't begin with . and survive intact.
4. Regex alternation matches in DECLARATION ORDER, not longest-leftmost. regex=CRON|CROND is a bug, CRON matches the prefix of CROND and leaves D dangling. Put longer alternatives first: regex=CROND|CRON|crond|cron. Applies to named patterns, inline {regex=...}, and any other alternation. When alternatives share a prefix, sort by length descending.
5. gron expands arrays into `[N]`-indexed attributes; index with `[N]`, NOT `.N.` Do not believe "gron drops arrays." $unmapped.{parse=gron}$ emits one queryable attribute per array leaf: unmapped.signInEventTypes[0], unmapped.additionalDetails[0].key, unmapped.targetResources[0].modifiedProperties[0].newValue. The trap that makes arrays look dropped is querying with dot-index (unmapped.targetResources.0.id → null); the real key is bracket-indexed (unmapped.targetResources[0].id). gron/dottedJson = each element its own attribute (use for "extract every field"); strict* = array kept as one JSON value for array_from_json(). Scalar/object flattening is identical across all three. Caveat: a raw PowerQuery columns/filter clause can't type the [, so validate [N] fields with powerquery_schema_discover, not a columns projection. To lift a whole array subtree to OCSF, rename_tree { from: "unmapped.targetResources[0].modifiedProperties", to: "entity.data" }. Tenant-validated 2026-06-22 (Microsoft Entra ID). See references/parse-directives.md.
6. `copy` first-of-N treats an empty string `""` as "exists" and lets it win. copy {from: ["userId","servicePrincipalId"]} where userId is "" copies the empty string, not the fallback, silently breaking identity-coalesce (e.g. service-principal / managed-identity sign-ins whose userPrincipalName/userId are empty but servicePrincipalName/servicePrincipalId hold the identity). Fix by dropping empty strings upstream (the collector/Lua serializer should omit "" values), or drop the empty source before the copy. Tenant-validated 2026-06-22.
7. On a LIVE source, a "still broken" event is usually pre-propagation, not a parser bug. Parser activation is ~3-5 min and a continuously-ingesting source keeps producing events through the OLD version during that window; SDL never re-parses historical events. Bump metadata.version every deploy and confirm via | group count() by metadata.version (or powerquery_schema_discover) which version parsed a given event before chasing a "bug." Related: if the parser sets event time from a log field, events are stamped at the log's own time, so widen validation query windows (24h/7d) or filter by a nonce field instead of time, or you get zero rows and a false negative.

When to use this skill

Use this skill when the user wants to turn raw log text into structured SDL events. Common triggers:

• "Write a parser for this <vendor> log."
• "Why isn't <field> showing up after ingest?"
• A pasted log line: CEF, syslog, JSON-per-line, key=value, multi-line, vendor-specific.
• Edits to an existing parser ("add a rewrite that masks the password", "tag every event with dataset=fortinet").
• Migration ("alias the new parser to the old one").

Do not use this skill for PowerQuery authoring (use powerquery) or for plain-text ingestion that does not need a parser (use sdl-api directly).

Workflow

Follow this loop. Skipping steps 1 (catalog check) and 5 (end-to-end validation) are the two most common reasons a parser goes live with surprises, so do not skip them.

1. Clear Gate 0 before anything else. Confirm the source's events carry both message and parser, so an SDL parser can actually run on them. If parser is absent, the fix is a DPM-set parser attribute, not a parser file. If neither attribute is present, parsing belongs in DPM and there is nothing to author here. See Gate 0 above and references/parser-eligibility.md.
2. Check the ai-siem catalog first. Before writing anything, search <https://github.com/Sentinel-One/ai-siem/tree/main/parsers> for the vendor/product. The repo has ~150 community + marketplace parsers; most common sources (Cloudflare, FortiGate, Palo Alto, Corelight, AWS RDS, Okta, Abnormal, Juniper, pfSense, Cisco ASA, etc.) are already there. If a parser exists, download it and diff against the user's requirements, far less work than authoring from scratch. See references/ai-siem-catalog.md for the recipe and a map of template parsers by shape.
3. Inspect the sample. Read every line the user pasted. Look for: leading priority/timestamp/host (syslog framing), the body shape (CEF | JSON | key=value | positional CSV | freeform), any obvious sub-structures (uri=..., embedded JSON, base64), and whether multiple physical lines belong to one logical event (stack traces, MySQL slow queries, Postgres SQL).
4. Decide on strategy. See Strategy decision tree below. Most logs are one of: alias an existing built-in, single line format with $field$ markers, repeating key/value catch-all, JSON parser with discardAttributes, line-grouper for multi-line, or a mapper for OCSF-style restructuring.
5. Draft the parser. Use the templates in examples/ as starting points, or (from step 1) a catalog parser as a base. Always start minimal, extract the outer frame first (timestamp + host + body), then add fragment formats or {parse=...} directives to crack the body open.

4a. Lint offline before you deploy. Run python3 scripts/lint_parser.py --new <parser-file> (standard library only, no tenant, no credentials). It checks the parser against the known traps in this file (key=value catch-all without its leading .* or repeat: true, a v0 constant keyed output, severity plus severity_id, same-name v1 renames, a catch-all mapping that is not last, stripped .* at either end of a regex, an anchorless lazy prefix, alternation order, root-only scratch discards, escaped dots, the mandatory attributes, claude_test_* names, and four shapes putFile rejects outright: a transformation entry with a second key such as _comment, an unsupported op such as regex, a {name=...} matcher option, and [] in a predicate; each rule was validated on a live tenant) and exits non-zero on any ERROR. Fix every ERROR before sdl_put_file. --rules lists the rules with the section each comes from. A clean lint means none of the KNOWN traps are present; it does not prove the parser parses, so step 5 is still the definition of done. Drop --new to audit an inherited catalog parser, where the hard rules report as findings instead of errors.

1. Deploy in place and validate on the live stream (DEFAULT). Update the SAME parser name the source already uses (/logParsers/<name>) via sdl_put_file with the current expectedVersion from sdl_get_file, bumping metadata.version on every deploy. Then validate against events the source is already sending: after ~3-5 min propagation, query dataSource.name='<Name>' metadata.version='<new>' and confirm the expected fields are populated and not null. Do NOT stand up a claude_test_* parser and do NOT synthetically HEC-ingest samples when the source is live, saved parser edits are non-destructive (they apply only to newly ingested events and never re-parse history), so editing the real parser is safe and the live feed is the authoritative test. The synthetic-ingest recipe in Validation is a FALLBACK, only for a source that is not currently ingesting. Before you deploy, commit the current live definition as a baseline, and after you deploy commit the new version, both to git in the local project folder only (see Validation → Version control & rollback). See Validation below. Do not call this skill done before the live query shows the fields populated.
2. Iterate. If a field is missing or wrong, identify which format/rewrite/mapping was responsible, edit, bump metadata.version, redeploy the same parser, wait for propagation, and re-query the live stream filtering on the new metadata.version. (Only for a non-live source: re-ingest the sample with a fresh nonce.)
3. Hand off. Show the user the final parser file (path + version), the sample they gave, and the parsed fields the query returned.

Required default attributes on every parser (MANDATORY)

Four attributes are MANDATORY on the top-level `attributes:` block of every parser, regardless of source. These are non-negotiable:

"dataSource.category": "security",    // ALWAYS hardcoded to "security": fixed constant, never change
"dataSource.name":     "Corelight",   // specific product/service name (example value)
"dataSource.vendor":   "Corelight",   // parent vendor / company name (example value)
"metadata.version":    "1.0.0"        // semver; required on every parser

These four are required by the downstream SDL pipeline (Marketplace routing, parser catalog, content-pack grouping). Omitting any of them produces a non-conformant parser even when field extraction is correct.

`dataSource.category` MUST always be hardcoded to `"security"`. This is not a taxonomy pick, it is a fixed constant. Never substitute another value (application, network, system, audit) regardless of the source type. Empirical audit of the ai-siem catalog (2026-04) shows ~17% of community parsers violate this rule with values like application, network, system, audit. Fix on copy.

dataSource.vendor is the parent company (Cisco for Umbrella, Microsoft for Azure AD). dataSource.name is the specific product (Cisco Umbrella, Azure AD). For single-product vendors both can be the same string (Corelight / Corelight).

metadata.version is semver (MAJOR.MINOR.PATCH with optional -rcN / -betaN suffix). Always increment on a new build so content-pack tooling can detect drift. Patch for fixes, minor for additive changes, major for breaking schema changes. May also be emitted from inside mappings via:

{ constant: { field: "metadata.version", value: "<your-current-version>" } }

A mappings.constant overrides the parser-root attributes: value when its predicate fires. This is the right pattern when one parser handles multiple vendor schema versions and you want each event labeled with the schema-version it actually came from.

Catalog-parser audit step. When you start from a catalog parser, the very first edits you make are to confirm all four fields are present, that dataSource.category is "security", and that metadata.version exists and reflects the change you're about to make. About 1/3 of the community parsers in Sentinel-One/ai-siem are missing dataSource.category outright; many that include it use a non-security value. Fix before doing anything else.

Default output schema: OCSF (mandatory reference)

Unless the user explicitly asks for vendor-native names, emit OCSF-shaped events: name captured fields with their OCSF dotted paths and tag every event with the class metadata (class_uid, class_name, category_uid, category_name, metadata.product.*, plus per-subtype activity_id on the format). OCSF is the mandatory default output schema for this skill.

OCSF mapping MUST always be done using `references/ocsf-schema-documentation.md`, the authoritative catalog (7 categories, ~25,759 field entries). Grep it for the exact dotted path before emitting any OCSF field name; never invent a name, never trust memory or a catalog parser. references/ocsf-mapping.md holds the OCSF class quick-picker (4001/4002/3002/1001, etc.), the field-to-dotted-path tables, and the two authoring idioms (capture-directly-into-dotted vs capture-vendor-native-then-rename). Consult both before writing any mapping.

Top-level parser structure

A parser file is augmented JSON (unquoted keys allowed, // and /* */ comments allowed). Top-level keys:

{
  timezone: "UTC",                                  // fallback for timestamp parsing
  attributes: {
    // REQUIRED defaults on every parser
    "metadata.version":    "1.0.0",
    "dataSource.category": "security",    // always "security", hardcoded, never change
    "dataSource.name":     "Juniper SRX",
    "dataSource.vendor":   "Juniper",
    // OCSF class metadata + vendor product metadata
    "metadata.product.vendor_name": "Juniper",
    "metadata.product.name":        "SRX",
    "metadata.log_provider":        "rt_flow",
    "class_uid":      4001,
    "class_name":     "Network Activity",
    "category_uid":   4,
    "category_name":  "Network Activity"
  },
  patterns:   { tsPattern: "\\w+\\s+\\d+\\s+[0-9:]+" },  // named regex shortcuts
  lineGroupers: [ { start: "...", continueThrough: "..." } ],  // multi-line glue
  formats: [
    {
      id: "happy-path",
      attributes: { activity_id: 1, activity_name: "Open" },  // per-subtype tagging
      format: "$time=tsPattern$ $src_endpoint.hostname$ $body$",
      discard: false,
      halt: true,                                   // stop after first match
      repeat: false                                 // re-apply until no progress
    }
  ],
  // Advanced (not always present):
  mappings: { mappings: [ /* rename vendor-native fields to OCSF dotted names */ ] }
}

formats is the primary engine. Everything else either feeds into it (patterns, lineGroupers) or runs after it (rewrites, mappings, discardAttributes).

Field-matcher syntax inside a format string

$fieldName=patternName{parse=...}{regex=...}{attrWhitelist=...}{attrBlacklist=...}{timezone=...}$

Rules that bite people often:

• $field$ alone uses the quotable pattern by default, which stops at whitespace or quote boundaries.
• $field{regex=\\d+}$ inlines a regex: backslash metacharacters double-escape (\\d not \d).
• Two adjacent $a$$b$ with no delimiter between them require a pattern or regex on $a$ so it knows when to stop.
• message is reserved: it's the raw event text. You cannot capture into message from a format. You can discardAttributes: ["message"] to drop it.
• A literal $ in the log requires \$$ (escape the dollar, then close the field).
• Any space in the format matches \s+. To match a single literal space use \\ (backslash-space).

For the full directive list see references/syntax.md and references/parse-directives.md.

Canonical reference parsers by input shape

Moved to a reference to keep SKILL.md under the 500-line limit. See `references/canonical-parsers.md`.

Strategy decision tree

Clear Gate 0 first: if the events do not carry both message and parser, no strategy below can run and the answer is DPM. Then apply the tree in order; first match wins:

1. Is there a parser in the ai-siem catalog? Search Sentinel-One/ai-siem for the vendor/product name. If so, start from that parser (see references/ai-siem-catalog.md for the fetch recipe and per-shape template map). On copy, immediately audit and fix:

• The 4 mandatory attributes (dataSource.category hardcoded to "security", dataSource.name, dataSource.vendor, metadata.version). About 1/3 of community parsers miss dataSource.category outright; many that include it use application/network/system/audit instead of "security". Force them to "security". Add metadata.version if missing, bump it if you're changing the parser.
• All OCSF field names against references/ocsf-schema-documentation.md. Catalog parsers frequently use vendor-native names (source_ip, dst_port) where OCSF dotted paths (src_endpoint.ip, dst_endpoint.port) are required. Rewrite via mappings.rename or change the capture name directly.
• Stale tenant-specific attributes (e.g. hardcoded site.id).
• class_uid should be an integer, not a string.
• Bump metadata.version.

1. Is there a built-in parser? Web access logs → accessLog. Pure JSON-per-line → json or dottedJson. Syslog → systemLog. Key=value pairs → keyValue. Heroku logplex → heroku-logplex. MySQL/Postgres → their dedicated parsers. CloudFront/ELB/S3/Redshift → AWS parsers. See references/builtin-parsers.md. If a built-in fits, recommend it and (optionally) just author an alias parser: { aliasTo: "json" }.
2. Is the log JSON-per-line but with a vendor envelope? Use one format that captures the envelope and applies {parse=json} (or dottedJson/escapedJson/urlEncodedJson/base64EncodedJson) on the embedded body. Add discardAttributes: ["message"] to save storage.
3. Is the whole line an OCSF/JSON blob you want flattened? Use the gron-capture-then-mappings idiom: format: "$unmapped.{parse=gron}$" at the top, then rename/copy/cast everything in a mappings block. See examples/08-gron-capture-template.json and the community/PARSER_TEMPLATE/ reference.
4. Is the body a sequence of `key=value` pairs? Use the two-format key/value idiom. The repeating catch-all format MUST be exactly ".*$_=identifier$=$_=quoteOrSpace$" (with the leading .*) and have repeat: true. Without the .* the format only anchors at position 0 of the message and fires zero times on any line that starts with a syslog priority <NNN>, a CEF header, an XML envelope, or anything else that isn't bare identifier=. The _ field name means "use the captured token as the field name". {parse=keyValue} is NOT a valid sub-parser directive; it returns Unknown parser "keyValue". The keyValue built-in can only be referenced via aliasTo: "keyValue" at the parser root (one-line alias parser), not via {parse=...}. Canonical references: built-in keyValue parser; examples/03-key-value.json; sentinelone/marketplace-fortinetfortigate-latest/ (FortiGate syslog, key=value with quoted strings).
5. Is it CEF or LEEF? Treat the header as a positional pipe-delimited line, then apply key/value catch-all to the extension. See examples/01-cef-over-syslog.json.
6. Is it positional / CSV / TSV? Use commaSeparatedValues / commaSeparatedvalues or pipeSeparatedValues parse directives with skipNumericConversion: true, and name the columns in a mappings block by positional index (attr[N]). Palo Alto Networks Firewall in ai-siem is the canonical reference.
7. Does the source emit multiple distinct event shapes (e.g. MySQL error vs MySQL general vs Postgres from one RDS stream)? Give each format an id: and fan mapping entries out with predicate: "<id>='true'". See the AWS RDS marketplace parser.
8. Are events multi-line? Add a lineGroupers block with start + one of continueThrough / continuePast / haltBefore / haltWith. Then write formats against the joined event. Beware HEC ingest newline splitting (see ingest-path gotchas). Canonical multi-line examples: parsers/community/sql_database_logs-latest/ (Postgres SQL) and parsers/sentinelone/marketplace-awsrdslogs-latest/ (RDS mysql/postgres mixed stream).
9. Is the source Windows Event Log XML? Use the per-EventID sub-parser idiom from parsers/community/microsoft_windows_eventlog-latest/: a lineGroupers block joins multi-line XML, then one format per EventID (4624, 4625, 4720, 4728, 1102, etc.) extracts EventID-specific fields. Remember the \\t / \\n double-escape gotcha. Class fan-out: 4624/4625 → class_uid: 3002 Authentication, 4720/4728 → class_uid: 3001 Account Change, 1102 → class_uid: 6004 Web Resources Activity (or whatever fits).
10. Do you need to restructure to OCSF or another schema? Use mappings (gron-style mappers). See references/mappers.md for the two equivalent syntaxes (v1 singular / v0 marketplace). For new parsers prefer v1; only stay on v0 when extending an existing v0 parser. Always look up dotted paths in `references/ocsf-schema-documentation.md` before emitting them.
11. None of the above? Hand-write a line format. Look for stable delimiters and use named patterns when delimiters are insufficient (e.g., a timestamp containing spaces). Drop events you never want (discard: true on the format) rather than letting them fall through to a looser format.

Common gotchas (memorize)

• Double-escape every regex metacharacter. \\d, \\s, \\., \\\\. The augmented-JSON layer eats one backslash before the regex engine sees it.
• `severity` is a RESERVED field coerced to 0-6 integer. The ingest pipeline maps info/warn/error strings to 0-6 using its own vocabulary. OCSF labels like "Informational" / "Medium" collide and get rewritten. For the OCSF string, emit severity_name (or any non-colliding name); for the integer, emit severity_id directly (int; the 0-6 range is validated, 99 "Other" is unvalidated). Never emit both severity and severity_id from the same parser. Tenant-validated April 2026.
• gron-captured dotted keys are FLAT, not nested. A source JSON key like "user.email": "alice" becomes unmapped.user.email as a single flat field, mappings must reference it with from: "unmapped.user.email" (no escaping). from: "unmapped.user\\.email" does NOT match on current tenants (re-confirmed live 2026-10-13; lint rule R13).
• Saved parsers apply only to newly ingested events. Historical logs are not re-parsed. So during iteration you must re-ingest the sample after every parser edit.
• A `parser=<name>`-tagged source with NO `/logParsers/<name>` file ingests raw (passthrough). If events carry parser='auditd' but dataSource.name / class_uid are null, the named parser file does not exist yet: the collector tags the events but SDL has nothing to apply. Deploy /logParsers/<name> and NEW events get parsed. After deploy, the parser's root attributes (e.g. dataSource.name) are the canary that the parser is RUNNING AT ALL, if they are still null on fresh events the parser has not propagated yet (not a format bug); if they are populated but a captured field is null, that IS a format/mapping bug. metadata.version is the canary for WHICH version is live: bump it on every deploy so you can tell from the live stream when a specific change has propagated (poll | group c=count() by metadata.version until the new version appears). Propagation observed at ~3-5 min on this tenant. Tenant-validated 2026-06-01.
• `halt: true` stops the format engine after a match. Use it for mutually exclusive formats (e.g., TCP vs UDP variants in pfSense). Without it, all matching formats merge their captures.
• `repeat: true` reapplies the same format from where it left off. Required for the key/value catch-all idiom, otherwise only the first pair is captured.
• `discardAttributes` MUST be per-format for scratch fields, NOT parser-root. A discardAttributes: [...] block at the parser root does NOT reliably drop fields created by named captures inside a format string ($_xxx{regex=...}$ scratch markers leak through to the event). Put discardAttributes: ["_pre", "_tail", ...] ON THE FORMAT OBJECT itself, alongside format:. The built-in json parser shows the shape: formats: [ { format: "$json{parse=json}$", repeat: true, discardAttributes: ["message"] } ]. Parser-root scope only works for fields produced by {parse=json} / {parse=dottedJson} flat expansion (vendor noise keys like flipkart_not_secure), not for format captures. Tenant-validated 2026-05-13.
• There is NO `from:` directive on a `format` entry. Only id, attributes, format, discard, halt, repeat, and rewrites are honored. A from: "<somefield>" next to format: is silently ignored, and the format then runs against the raw line; a greedy regex like [\s\S]+ will clobber every targeted field with the entire log message on every event. The from: key only exists in mappings ops ({rename: {from, to}}, {copy: {from, to}}). To run a format-like rewrite against a specific captured field, use rewrites: on the same format with input: "<fieldname>", or do the work in a mappings op.
• The parser sees only `message` and `parser`. parser selects which parser runs; message is the content it runs against (see Gate 0). Every other attribute is invisible to it. Ingest-time attributes (app_id, cluster, region, custom enrichments from collectors / HEC ingest) are NOT visible to format strings, format-level discard: filters, or rewrites regex inputs. Those ingest-time attributes ARE visible to mappings predicates and mappings op from: paths since mappings run after parsing. Fields already promoted to top level by an upstream pipeline are unreachable from a format for the same reason, so transform those in DPM. Encode any sentinel you need to gate a format on into the message body or anchor on a string the message itself contains.
• v1 `mappings` is FIRST-MATCH-WINS. Mapping entries are tried in declaration order; only the first entry whose predicate matches an event runs. A predicate: "true" catch-all therefore MUST go LAST in the list, placed first, it shadows every per-app/per-class entry below it (renames don't apply, class constants stay generic). Pattern: most specific predicates first, generic catch-all last. If two predicates would both fire on the same event and you want both effects, fold their transformations into a single entry. Tenant-validated April 2026.
• Fragment-format anchors must live INSIDE the prefix's regex, not as separate format-string text. The format $_pre{regex=[\s\S]*}$/actuator/health$_suf{regex=[\s\S]*}$ (greedy [\s\S]* followed by literal text in the format string between the field markers) does NOT match, the engine commits each field's regex independently and won't backtrack across literal-text gaps. Working form: $_pre{regex=[\s\S]*\/actuator\/health}$$_suf{regex=[\s\S]*}$ (literal anchor inside the prefix's regex). Same shape as Phase-10 IOC formats ([\s\S]*X-Forwarded-For=[\[]? baked into the prefix).
• Greedy `[\s\S]*ANCHOR` lands on the LAST occurrence of `ANCHOR`. That is the right behaviour when the captured token follows the rightmost anchor on the line (Dropwizard [\s\S]*\[dw-[0-9]+ - is unique to the request-line bracket). It fails when the anchor is too generic: [\s\S]*\[ followed by a UT-[0-9]+ capture fails on any line whose last [ is something else (e.g. [#033[36m...] ANSI brackets). Two fixes: (a) make the anchor multi-character and discriminating ([\s\S]*\[UT-), accepting that the literal is consumed by the anchor and your capture holds only the trailing portion ($user_task_id_num{regex=[0-9]+}$); or (b) use a lazy quantifier ([\s\S]*?) plus a discriminating literal when the FIRST occurrence is the right one.
• Greedy `[\\s\\S]*<anchor>` collides with COMPOUND field names that contain the anchor as a substring. Anchoring a record-type capture on bare type= ($_pre{regex=[\\s\\S]*type=}$$rectype{regex=[A-Z0-9_]+}$) silently mis-fires on auditd PATH records because nametype=NORMAL (and objtype=) contain type=, the greedy match lands on the LAST type= (the one inside nametype=), captures NORMAL, and the rest of the format then fails. Fix: widen the anchor with the delimiter that precedes the REAL token, e.g. type= with a leading space ($_pre{regex=[\\s\\S]* type=}$), since the real record-type follows a space (audispd: type=PATH) whereas nametype=/objtype= are preceded by a letter. General rule: when your anchor is a short substring that also appears inside longer field names, disambiguate it with a neighbouring delimiter. Tenant-validated 2026-06-01 on a Linux auditd parser: bare type= parsed SYSCALL/EXECVE/PROCTITLE but dropped every PATH record; type= fixed all record types.
• Per-event parse limit is just under 50,000 chars. Long events get truncated before parsing.
• Line groupers max 100,000 chars per joined event. Above that, the grouper still emits but truncated.
• `intermittentTimestamps: true` is required for logs (like MySQL general query log) that emit a timestamp only on the second they roll over. Without it, every line missing a timestamp gets the ingest time.
• Regex alternation cannot wrap a `$...$` token. ($a$|$b$) is invalid. Use multiple line fragments instead.
• SDL regex engine does NOT support lookarounds. (?=...), (?!...), (?<=...), (?<!...) all fail silently. Use explicit token delimiters or split into multiple formats.
• SDL strips the leading `.*` from a regex before passing it to the Java engine. This means using .*? (non-greedy) at the start of a regex produces an orphaned ? which the engine rejects as "Dangling meta character '?' near index 0". Use plain .* and rely on normal greedy matching. If you need non-greedy behaviour, restructure the capture into two segments (see below).
• Trailing `.*` at the END of a format string is also stripped (same root cause as leading `.*`). A format ending in $_body{regex=.*}$ or $process.cmd_line{regex=.*}$ (the field marker is the last thing on the line, regex is exactly .*) silently captures the EMPTY STRING instead of "everything to end of line". The SDL leading-.*-strip rule applies regardless of whether the .* is the entire regex or merely starts it; with the .* stripped, the regex becomes empty and the field defaults to a zero-length capture. Tenant-validated 2026-05-26 on a linuxLogsParser sudo format: COMMAND=$process.cmd_line{regex=.*}$ captured "" (empty), whereas the same format with regex=[\\s\\S]+ captured the full command-line including embedded spaces. Use [\\s\\S]+ (one or more, including newlines) or [\\s\\S]* (zero or more, including newlines) at any field marker where you want "everything that remains on the line" semantics. Both forms avoid the leading-.* strip and survive intact.
• Regex alternation matches in DECLARATION ORDER, not longest-leftmost. regex=CRON|CROND is a bug: CRON is tried first and matches the literal prefix of CROND, leaving D to fail against the next literal in the format string. The fix is to put the LONGER alternative FIRST: regex=CROND|CRON|crond|cron. This applies anywhere alternation appears (named patterns, inline {regex=...}, regex in rewrites). Tenant-validated 2026-05-26: linuxLogsParser cron_cmd format with CRON|CROND matched zero CROND events; same format with CROND|CRON matched every cron variant. General rule: when alternatives share a prefix, sort by length descending.
• `[\s\S]*?` (lazy) at the start of a prefix regex silently matches almost nothing, same family of bugs as `.*?`, but no error. A fragment format like $_pre{regex=[\s\S]*?}$$field{regex=PATTERN}$$_suf{regex=[\s\S]*}$ will fire on a tiny fraction of events instead of the expected matches, because the SDL regex engine does not backtrack across field boundaries when the prefix is lazy. The fix is the same shape as the multi-char-anchor rule: use greedy [\s\S]* plus a discriminating literal anchor inside the prefix regex itself (e.g. [\s\S]*\"merchant_transaction_id\":\" to anchor on a JSON key before capturing a CL transaction id). Tenant-validated 2026-05-13. If you actually need first-match semantics, anchor on a literal that only appears once per line (e.g. \[log\., \"url\":\"), do not reach for *?. Live-validated 2026-10-13: the anchorless form ($_pre{regex=[\s\S]*?}$) makes the whole format fail to match, while a lazy prefix WITH a literal anchor inside its own regex ([\s\S]*?COMMAND=) captures correctly.
• When no format matches an event, the parser's root `attributes` are not applied either (live-validated 2026-10-13 on three independent failure shapes). A null dataSource.name on fresh events therefore means "no format matched" as often as "the parser has not propagated": check one before assuming the other.
• Two-segment sequential capture pattern for JSON values inside a raw string. When a log line is a raw JSON string (with \" escaping) and you need to extract a field value, use a two-segment format with a $$ transition. First segment grabs everything up to and including the opening quote of the value into a scratch field; second segment captures the value up to the closing quote. Example for extracting severity_name before SDL coerces it: $_ps{regex=.*\\\"severity\\\":\\\"}$$severity_name{regex=[^\\\"]*}$. The $$ boundary: the first $ closes seg 1, the second $ opens seg 2. The scratch field (_ps) must be in the format's OWN discardAttributes: ["_ps"] array (per-format, alongside format:), parser-root discardAttributes does not reliably drop format-capture scratch fields on this tenant. This pattern works for any quoted string value in a JSON-encoded log line.
• `$pri{parse=syslogPriority}$` can cause silent event drops on some tenants. If you suspect priority parsing is the culprit, capture the priority as a plain field ($pri{regex=\\d+}$) and skip the parse directive.
• Indexed positional access `attr[N]` only works in `mappings.version: 1`. v0 mappings reject attr[0]/attr[1] syntax. If the source is positional CSV (Palo Alto Firewall is the canonical case), use v1. Reference: parsers/sentinelone/marketplace-paloaltonetworksfirewall-latest/.
• `{parse=gron}` flattens nested JSON to dotted keys, but the keys themselves contain dots. A source {"Resource": {"Id": 5}} becomes the single flat field unmapped.Resource.Id (one key with two dots), not a nested object. To rename it use from: "unmapped.Resource.Id" (no escapes). To rename a whole subtree use rename_tree: { from: "unmapped.Resource", to: "resource" } and the engine walks the dotted-prefix subtree. Reference: parsers/sentinelone/marketplace-cloudflare-latest/.
• `attrBlacklist={field1,field2}` only filters subfields produced by a `{parse=...}` directive. It does NOT filter top-level captured fields you named explicitly in the format string. To drop a top-level field use discardAttributes: ["fieldname"] at the parser root. Reference: parsers/sentinelone/marketplace-fortinetfortigate-latest/.
• `{parse=...}` subfield naming is `<prefix><CapitalizedFirstLetterOfKey><restOfKey>` (per SDL KB 000006743). Field details parsing ?apple=w&banana=x yields detailsApple and detailsBanana, NOT details_apple / detailsQuery_apple / details.apple. Same convention for {parse=json} (camelCase): details parsing {"foo":"hello"} produces detailsFoo, not details_foo. Confirm the exact field name with a sample query if unsure, every "obvious" alternative spelling fails silently.
• `{parse=uri}` and `{parse=uriAttributes}` consume the parent capture instead of preserving it. After parsing, the original captured value is gone from the event; only the generated subfields remain (validated against current tenant April 2026). If you also need the raw URL/query string, capture it twice: once with the parse directive, once into a sibling scratch field, and rename the scratch one in mappings. Or split path/query directly in the format: $path{regex=[^? ]+}$\??$qs{regex=[^ ]*}{parse=uriAttributes}$.
• `{parse=uri}` on relative URLs (no scheme/host) silently drops the query subfields on tested tenants. It still emits <prefix>Path, but <prefix><Key> siblings for query parameters never materialize. Use {parse=uriAttributes} on the query-string portion (split manually with regex) instead; it works regardless of whether the URL is absolute.
• `skipNumericConversion: true` is required when columns contain numeric strings you want preserved as strings. Without it, commaSeparatedvalues and pipeSeparatedValues parse will coerce "00123" → 123 (loses leading zeros), "0xDEADBEEF" → 3735928559, and "1.0" → 1. Set skipNumericConversion: true on the format whenever any column is an opaque ID or a hex/octal string. Canonical: Palo Alto Firewall.
• `intermittentTimestamps: true` is the only fix for sources where only the first record on each second has a timestamp. Common with MySQL general query log, AWS RDS, sometimes VPC flow logs. Without it, every line that lacks a timestamp gets the ingest timestamp, not the inferred one. Reference: parsers/sentinelone/marketplace-awsvpcflowlogs-latest/ and parsers/community/sql_database_logs-latest/.
• `enum` cast in v1 mappings without `enum_default` leaves unmapped values UNCHANGED, not null. If you want unknowns to map to a known sentinel, set enum_default: <value> on the cast. Cloudflare uses enum_default: 99 ("Unknown") consistently, copy that pattern.
• `severity_id` takes the OCSF integer enum; `severity` is RESERVED and coerced by ingest. Setting severity_id: 999 silently fails; the 0-6 range is validated, 99 ("Other") is unvalidated. Emit severity_id (int) plus optionally severity_name (the OCSF label string). Never emit severity, and never emit BOTH severity and severity_id from the same parser.
• Rewrites run during format parsing; mappings run after all formats. A field renamed in mappings.rename cannot be referenced by a rewrites: block on a later format on the same line because rewrites: runs per-format during capture. If you need a derived field from a renamed one, do the derivation inside mappings (use copy then cast, or replace).
• Mixing `mappings.version: 0` and `mappings.version: 1` syntax in the same block is a hard error. The engine rejects with "expected list got object" or "unsupported event mapper version -1". Pick one version per parser and stick to it.
• Windows Event Log XML payloads ship with double-escaped tabs/newlines. A parser for Windows EventLog in the catalog (parsers/community/microsoft_windows_eventlog-latest/) deals with \\t and \\n literals embedded inside XML strings; capturing them needs \\\\t / \\\\n (four backslashes) at the augmented-JSON layer.
• Some catalog parsers ship without `dataSource.category` at all (~33% of community). When you copy from Sentinel-One/ai-siem, audit the top-level attributes: block. If dataSource.category is missing, add it as "security". If present with another value, change it to "security". Same audit for dataSource.name and dataSource.vendor.
• `$_=identifier$=$_=quoteOrSpace$` repeating idiom REQUIRES a leading `.*` AND `repeat: true`. This is the canonical built-in keyValue parser shape and it is documented in the public KB as ".*$_=identifier$=$_=quoteOrSpace$". Without the leading .*, the format anchors at position 0 of the message and fires zero times on any line that doesn't start with an identifier, including every syslog log (<189>date=...), CEF (CEF:0|...), Windows Event (<Event xmlns=...>), or anything with a leading timestamp/priority. The format extracts nothing in that case, silently. The .* lets the engine scan forward to the first key= and then repeat: true walks the remaining pairs from there. Tenant-validated 2026-05-26 on FortiGate syslog (<189>date=2026-05-25 time=18:07:18 devname="..." ...): without .* zero fields captured; with .* all 30+ key=value pairs captured. The keyValue built-in parser uses exactly this shape (see examples/03-key-value.json). If you also need a separate static-prefix capture (e.g. to extract a syslog priority), put it in a SIBLING format that runs before the catch-all, each format runs against the full original message independently, so the prefix format and the .*-anchored KV format don't compete.
• Conditional class assignment (one parser, multiple OCSF classes) is best done by stamping `class_uid`/`class_name` on each `format`'s `attributes:` block, not on the parser root. The root attributes are baseline; per-format attributes override. Reference for fan-out via format-id sentinel + mapping predicates: parsers/sentinelone/marketplace-awsrdslogs-latest/.
• Same-name `rename` in v1 mappings silently NULLS the field. Writing { rename: { from: "action", to: "action" } } to "keep" a captured raw field under its own name actually deletes the value. The KV catch-all extracts action (or any other raw vendor field) directly into the event under its native name, if you want to keep it, OMIT the rename. Only write a rename when the target name is different from the source. Tenant-validated 2026-05-26: identical FortiGate action="perf-stats" event returned action=null with the self-rename present, action="perf-stats" after removing the self-rename. This bites you when "promoting" raw fields that the catch-all already captures.
• `raw_data` JSON blob is duplicate storage. Many upstream Observo / collector extractions leave the entire raw event in a raw_data field alongside the typed-OCSF fields they extract. If your parser is also extracting those same fields via KV catch-all or named captures, the raw_data field is redundant and storage-wasteful. Either discardAttributes: ["raw_data"] at parser root or strip upstream. Don't store the same key=value pair three times (typed + unmapped + raw_data); this was the Netskope anti-pattern. Audit on copy.
• Unquoted numerics captured by `quoteOrSpace` are auto-typed as NUMBER columns. No explicit cast required. A FortiGate srcport=44321 dstport=443 sentbyte=8421 line going through .*$_=identifier$=$_=quoteOrSpace$ produces srcport: 44321 (number), dstport: 443 (number), sentbyte: 8421 (number), all queryable with avg(), sum(), > 100, etc. without number() wrapping. The auto-inference fires when the captured token is purely numeric AND unquoted in the source. Quoted values (level="notice") stay as strings even if they look numeric. Tenant-validated 2026-05-26 on FortiGate. Caveat: SDL's column type-locking still applies, if a column was ingested as string earlier, new numeric writes coerce back to string and avg() returns NaN. The CLAUDE.md project rule "wrap with number() before arithmetic" is still the bulletproof pattern for dashboard panels and STAR rule bodies, especially for counter columns (traffic.bytes_*, traffic.packets_*, duration).
• `message` field is reserved and preserved on every event regardless of parser activity. SDL never lets a parser overwrite message; the raw event body always survives. Practical consequence: dashboards and PowerQuery hunts that use parse "fieldname=$x$ " from message continue to work even after a parser renames the typed columns (srcip → src_endpoint.ip). This is the backward-compat escape hatch during parser rollout, old events still parse via message-text extraction, new events parse via typed columns. To migrate a dashboard from parse-from-message to typed columns without losing coverage on legacy events, use (typed_field == * OR raw_field == *) or coalesce(typed_field, raw_field) patterns. Tenant-validated 2026-05-26 on FortiGate dashboard rewrite: panels that switched to src_endpoint.ip continued returning the same row counts because the upstream Observo path was already producing typed src_endpoint.ip too; panels using tls.sni only show fortigateParser-tagged events because Observo didn't extract SNI.
• When two ingest paths produce OCSF from the same source, they may flatten `event.type` / `event.subtype` differently. Common pattern: upstream Observo collector flattens FortiGate type=traffic subtype=forward into a single event.type='forward' (drops the parent type). The downstream parser keeps both: event.type='traffic', event.subtype='forward'. Same source, two schema shapes coexisting on different events. Dashboards and hunts that filter on subtype must use OR: (event.subtype='forward' OR event.type='forward'); this matches both legacy and parser-tagged events. The rule: every time you migrate a downstream consumer to typed OCSF columns, audit each subtype filter for backward compat with the upstream shape during the rollover window.
• `parser='<name>'` field may only be tagged for very recent events. Some tenants are configured to stamp the parser field only on events ingested in the last N minutes (e.g., last 30 minutes). Queries filtering on parser='myparser' against a 24h window return only that recent slice, not 24h of data. For broad coverage validation, filter on dataSource.name + metadata.version (which IS retained per-event) instead, and use parser= only for isolating a specific recent ingest test. Tenant-validated 2026-05-26.

Ingest-path gotchas (discovered empirically via live validation)

These are quirks of the ingest pipeline itself, not the parser DSL. They bite you at validation time even when the parser JSON is syntactically perfect.

• HEC ingest splits on `\n` BEFORE the parser runs. Multi-line events (stack traces, MySQL slow queries) get chopped up and lineGroupers cannot re-fold them. Workarounds:

1. Fold multi-line events client-side into a single line with a sentinel character (e.g. \x1e), then have your parser split on that sentinel.
2. Send each logical event as a single HEC event so newlines inside the event body are preserved.

• The `server-host` upload header is unreliable for isolating a test. SDL sometimes overrides the header to the literal source name, and if the parser extracts a host field from the log itself that wins too. Do NOT filter your validation query by host='parser-test-<uuid>' alone. Safer: filter by parser='claude_test_<name>' and _bytes > 0, and use a unique nonce in the payload to double-check isolation.
• `{parse=dottedJson}` prefixes subfields when the field has a non-empty name. $payload{parse=dottedJson}$ on {"user":"alice"} yields payload.user = "alice". If you want top-level fields, capture into an empty name: ${parse=dottedJson}$. Same applies to json, escapedJson, urlEncodedJson, base64EncodedJson.
• Existence checks: `config_file(name=...)` returns `None` for a missing path, it does not raise. The legacy REST get_file raises HTTP 404 instead of returning success=false with noSuchFile. Prefer the GraphQL method and test for None. Parsers live at /logParsers/ and are name-addressed, so no udoId is involved.

mappings block gotchas

Two syntaxes coexist on live tenants and both work. You must commit to one per parser, mixing them in the same mappings block produces confusing "expected list got object" errors.

v1 (singular, preferred for new parsers, tenant-validated April 2026):

• `mappings.version: 1` is mandatory. Without it the tenant returns unsupported event mapper version -1.
• Each transformation is `{<op>: {...body}}`, where the op name is the outer key (e.g. {rename: {from, to}}). The public-doc form {op: "rename", from: ..., to: ...} is rejected.
• `rename.from` is a single string, not a list. Use copy (which does accept a list) if you need "first of N".
• `cast` uses `type:`, not to:. Example: {cast: {field: "x", type: "int"}}.
• `cast` overwrites the source field. There is no output: parameter, copy first, then cast the copy.
• Predicates use `==` (double equals).

v0 (plural-grouped, used by most S1 Marketplace parsers, AWS RDS, Corelight, Cloudflare, FortiGate):

• `mappings.version: 0`. Same block, different body shape.
• Ops are plural arrays on the mapping entry: renames: [...], copies: [...], constants: [...]. No transformations: wrapper.
• Every op entry uses `inputs: [list]` + `output:` + `type:`: even rename. type: "string" is the safe default.
• Predicates use `=` (single equals) in v0, both on the mapping entry and inside constants.
• v0 `constants:` entries are keyed `field`, never `output`. [{output, value, type}] is rejected by putFile with Expected a field; [{field, value, type}] deploys and the constant is applied to events. Live-validated 2026-10-13 (the 2026-05-26 note that every v0 constants: block is rejected was the output shape). scripts/lint_parser.py rule R03 flags the output shape.

If you inherit a marketplace parser, keep it on v0 rather than rewriting. For new parsers, pick v1. See references/mappers.md for the full side-by-side.

`constant` / `constants` op (both versions) is the workhorse for conditional OCSF class/activity assignment, give it a predicate and it only fires when matched, so you can fan one vendor-native state code (conn_state, action, etc.) out into activity_id + activity_name + type_uid triples.

Inline `predicate:` on individual `constant` ops is the cleanest escape hatch from first-match-wins in v1. When you want a long list of unconditional renames/copies AND a handful of conditional constant writes (severity bucketing, OCSF enum derivation from a vendor string, action/status/disposition fan-out), DO NOT split them across multiple top-level mapping entries, only the first matching entry fires. Instead, put EVERYTHING in one predicate: "true" entry and attach a predicate: field directly to each conditional constant op:

mappings: {
  version: 1,
  mappings: [{
    predicate: "true",
    transformations: [
      // ... unconditional renames / copies / casts ...
      { copy: { from: "raw_severity", to: "vendor.severity_raw" } },
      // Conditional constants: predicate lives ON THE OP, not the mapping entry.
      { constant: { field: "severity_id", value: 1, predicate: "vendor.severity_raw >= 0 && vendor.severity_raw <= 3"  } },
      { constant: { field: "severity_id", value: 3, predicate: "vendor.severity_raw >= 4 && vendor.severity_raw <= 6"  } },
      { constant: { field: "severity_id", value: 4, predicate: "vendor.severity_raw >= 7 && vendor.severity_raw <= 8"  } },
      { constant: { field: "severity_id", value: 5, predicate: "vendor.severity_raw >= 9 && vendor.severity_raw <= 10" } },
      { constant: { field: "action",    value: "Allowed", predicate: "raw_action == 'alert' || raw_action == 'allow'" } },
      { constant: { field: "action_id", value: 1,         predicate: "raw_action == 'alert' || raw_action == 'allow'" } },
      // ... etc ...
    ]
  }]
}

This pattern preserves first-match-wins semantics at the mapping-entry level (which can't be worked around) while still letting you express N independent conditional writes without duplicating the rename list across N per-subtype blocks. Use it whenever you have a flat schema (one event shape, many conditional derivations), severity bucketing, action/status/disposition fan-out from a vendor enum, HTTP-status → OCSF status_id, and so on. The alternative, splitting into per-subtype top-level mapping entries, only makes sense when the per-subtype rename lists are genuinely different (different OCSF classes per event subtype). Tenant-validated May 2026 on a CEF source with ~20 conditional ops: all fired from a single predicate: "true" mapping entry; the same logic split across multiple top-level entries produced zero hits on every conditional write because only the first entry ran. See examples/11-*-cef-ocsf.json for the runnable reference.

Predicate type-mismatch silently fails on captured numerics. The KV catch-all auto-types unquoted values as NUMBER (pitfall #5 in Critical pitfalls). That means a predicate like raw_severity == '5', quoted string; never matches a raw_severity captured as integer 5. Use bare numerics in predicates for numeric fields (raw_severity >= 4, http_response.code < 400) and quoted strings only for actually-string fields (raw_action == 'alert', format-id sentinels mySqlErrorLog == 'true'). When in doubt, run a quick query on a recent parsed event and check cellType in the response, STRING vs NUMBER, before writing the predicate. A string predicate against a numeric field silently drops every dependent op; the only symptom is the target field being null on every event, which can take several deploy/iterate cycles to localize. Tenant-validated May 2026.

OCSF `severity_id` is an enum `{0,1,2,3,4,5,6,99}`, NOT a vendor passthrough (0-6 is tenant-validated; 99 is unvalidated). Many sources (CEF 0-10, syslog 0-7, custom 0-100) ship a wider-range numeric severity. Renaming the raw value directly into severity_id produces invalid OCSF (e.g. severity_id=10 when a vendor uses 0-10) AND semantically false detections (vendor "5 out of 10" means Medium for that vendor, but severity_id=5 is OCSF "Critical", every routine event suddenly looks Critical to downstream consumers). Always bucket via per-op predicates and preserve the raw value under a vendor namespace (vendor.severity_raw, cef.severity) for source fidelity. Reference bucketing for a 0-10 scale: 0-3 → 1 Informational, 4-6 → 3 Medium, 7-8 → 4 High, 9-10 → 5 Critical. For 0-7 syslog: 0-1 → 5 Critical, 2-3 → 4 High, 4 → 3 Medium, 5-7 → 1 Informational. Same severity (no suffix) is RESERVED and coerced to 0-6 by ingest, so emit severity_id (the integer) and optionally severity_name (a label string), never both severity and severity_id from the same parser. Tenant-validated May 2026: a parser that passed CEF 0-10 directly into severity_id emitted invalid OCSF values (severity_id=10) AND tagged the bulk of routine traffic as Critical; the bucketed replacement produced semantically correct severity_ids for the same input stream.

Format-id-as-sentinel predicate: format: { id: "mySqlErrorLog", ... } auto-sets a boolean-string field mySqlErrorLog='true' on events that matched that format. Use predicate: "mySqlErrorLog='true'" (v0) or "mySqlErrorLog == 'true'" (v1) to fan mapping entries out to sub-shapes. Quote the 'true'; it's a string, not a bare boolean.

`mappings.version: 1` uses FIRST-MATCH-WINS, not all-match. The first mapping block whose predicate matches consumes the event; no subsequent blocks run. This is the single most important structural constraint when using v1. Consequences:

• A predicate: "true" unconditional block placed FIRST will fire on every event and prevent all per-app conditional blocks from ever executing. Always place the catch-all at the END.
• drop / drop_tree transformations cannot be factored into a standalone unconditional block at the start and shared across per-app blocks. They must be duplicated into each per-app block. Verbose but correct.
• `rename` ops have the SAME problem as `drop`. If you have a catch-all rename block at the end (typed-OCSF migration: srcip → src_endpoint.ip, devname → device.name, etc.) AND per-subtype constant blocks earlier (for activity_id fan-out), the catch-all rename NEVER fires for events that matched a per-subtype block, only the catch-all events get renamed. The per-subtype events keep raw vendor field names. To make the parser produce typed OCSF on ALL events, you MUST duplicate the rename list into every per-subtype block alongside its constants. Tenant-validated 2026-05-26 on FortiGate v1.1.0: with renames only in catch-all, event.subtype='forward' events had activity_id=6 from the constant block but src_endpoint.ip=null because the rename never ran; with renames duplicated into the per-subtype block, all fields populated. Pattern: each per-subtype block = its own constants + the full rename list. Verbose but correct.
• v0 mappings DO run all matching blocks (marketplace parsers like Cloudflare and FortiGate rely on this: an unconditional first block handles common renames, then conditional blocks handle sub-types). If you need multiple pass-through effects, consider whether v0 is a better fit, but note v0 lacks drop/drop_tree AND v0's constants: is currently rejected on this tenant (see WARNING above), so v0 is only viable if you don't need either.

`discardAttributes` MUST be placed per-format, not at parser root, for scratch fields created by format captures. The parser-root discardAttributes: [...] block does NOT reliably drop fields that named-capture markers $_xxx{...}$ produced inside a format string. Scratch fields like _dw_pre, _dw_tail, _lc_pre, _lc_suf, etc. leak through to the final event despite being listed at the root. The reliable form is to put discardAttributes: ["_dw_pre", "_dw_tail", ...] ON THE FORMAT OBJECT ITSELF, as a sibling key to format:. This matches the built-in json parser's shape: formats: [ { format: "$json{parse=json}$", repeat: true, discardAttributes: ["message"] } ]. Tenant-validated 2026-05-13. Parser-root discardAttributes can still be used for noise fields produced by {parse=json} / {parse=dottedJson} flat-expansion of vendor JSON keys (e.g. flipkart_not_secure, upstream_addr); those are not format captures and do get dropped from the root.

`discardAttributes` also does NOT remove fields produced by `{parse=gron}` at the top level. If {parse=gron} extracts host, facility, sndr, etc. directly to the event root, adding them to discardAttributes (root OR per-format) has zero effect. Use mappings.drop / mappings.drop_tree for those instead (see the next two gotchas). Tenant-validated May 2026.

`attrBlacklist` on a bare `{parse=gron}` format cannot drop top-level fields. The attrBlacklist directive only filters subfields when a NAMED PREFIX is in use, e.g. $payload{parse=gron}{attrBlacklist=debug}$ drops payload.debug. Without a named prefix (${parse=gron}{attrBlacklist=host,sndr}$), the blacklist is ignored for top-level keys because there is no prefix namespace to filter within. This means {parse=gron} with attrBlacklist cannot be used to suppress noise fields at the root level. Tenant-validated May 2026.

The correct mechanism for dropping top-level gron-produced fields is `mappings.drop` / `mappings.drop_tree`. Add { drop: { field: "host" } } (scalars) and { drop_tree: { field: "object" } } (subtrees) in the transformations of every applicable mapping block. Because v1 is first-match-wins, these drop ops must be duplicated into each per-app block AND in the catch-all block; there is no shortcut.

computeFields gotchas

• Use PowerQuery ternary `a ? b : c`, not if(a, b, c). The latter returns Unknown function 'if'.
• Use `==` for equality, not =.
• Cannot reference dashed field names. protocol-id is parsed as subtraction. Rename via mappings first, or do the translation entirely in the mappings block.
• Place the rewrite on the format that captures the source field. If a repeating key/value sub-format is what produces the field, the rewrite belongs on the sub-format, not on the frame format above it.
• The `replace` MAPPER op is a runtime no-op on this tenant; use a `computeFields` + PowerQuery `replace()` rewrite instead. Two traps with { replace: {...} } in a mappings block: (1) the regex key is regexp, NOT pattern: pattern returns 400: Missing required key 'regexp' on putFile; (2) even with the correct regexp key the op silently does NOT transform the value at ingest (validated on both a dotted target process.cmd_line and a flat scratch field, 2026-06-01). The working substitute is a computeFields rewrite calling the PowerQuery replace(field, regex, replacement) string function. PowerQuery string literals are SINGLE-quoted, so a regex containing double-quotes needs no escaping inside the expression: expression: "| let _cmdline = replace(_cmdline, 'a[0-9]+=\"([^\"]*)\"', '$1')". Capture into a FLAT scratch field (computeFields and replace address flat names cleanly; dotted paths are unreliable), then rename the scratch field to the dotted OCSF target in mappings. This is how the auditd parser turns an EXECVE arg vector a0="cp" a1="/x" a2="/y" into a readable process.cmd_line = cp /x /y.
• There is NO hex / base16 decode primitive anywhere in the DSL or in `computeFields`. The {parse=...} directives offer base64EncodedJson / urlEncodedJson but no hex; the PowerQuery subset available to computeFields has no position-aware hex decoder either. auditd proctitle (and individual hex-encoded EXECVE args) are NUL-separated hex and cannot be turned into ASCII in-parser. Source the readable command line from the EXECVE arg vector instead (auditd stores those as plain ASCII), and leave the hex proctitle as-is. Do not burn iterations trying replace/regex to hex-decode.

Interaction between halt: true and repeat formats

• `halt: true` stops ALL further format matching on that line: including repeating key/value catch-alls. If you have a two-format pattern (frame + repeating extractor), do NOT put halt: true on the frame format; the extractor will never fire.

Validation (mandatory)

Always validate against the live tenant, the only authoritative test is "did the field actually appear in a query after the parser ran." Syntactic plausibility of the JSON is not validation.

Read-only mode. From 1.5.5, s1-secops-mcp is read-only by default (S1_READ_ONLY unset or true). sdl_put_file, sdl_delete_file and hec_ingest are refused in that mode, so live validation needs write; sdl_get_file, sdl_list_files and every validation query still run. Check the mode before you start a put-ingest-query loop: read mode.readOnly from the sentinelone://credentials-status resource, or the server's initialize instructions.

• If a write returns "s1-secops-mcp is in read-only mode (S1_READ_ONLY).", stop. Do not retry, and do not route around it with another tool (s1_api_post to a different endpoint, a hand-built GraphQL mutation).
• Tell the user which action was refused and how to enable write: Docker launcher s1-secops-mcp-launch.sh install --read-write (Windows -ReadWrite), then restart Claude Desktop; .mcpb extension: turn off "Read-only mode (recommended)"; Node install: "S1_READ_ONLY": "false" in the server's env block. Detail: docs/read-only.md in the source repo (not shipped).
• Offer the parser (and the sample log lines) as local files via the Write tool, and state plainly that the parser is unvalidated until it runs on the tenant.
• Never enable write on the user's behalf by editing their MCP client config.

Default: update the real parser in place, validate on the live stream

When the source is already ingesting (the normal case), edit the real parser and validate on real traffic. Do NOT create a claude_test_* parser and do NOT synthetically ingest, saved parser edits are non-destructive (they apply only to newly ingested events and never re-parse history), so editing the live parser is safe, and the feed the source is already sending is the authoritative test.

MCP tool calls (s1-secops-mcp), in order:
 
1. sdl_get_file   {path: "/logParsers/<name>"}
   -> note the returned version (optimistic locking + rollback reference)
 
2. sdl_put_file   {path: "/logParsers/<name>", content: <parser body>, expectedVersion: <version>}
   -> deploy the edit to the SAME path; bump metadata.version in the file first
 
3. Wait ~3-5 min for propagation (activation is NOT instant on this tenant).
 
4. powerquery_run {query: "dataSource.name='<Name>' metadata.version='<new>' | group count=count() by <subtype-field>", hours: 1}
   powerquery_run {query: "dataSource.name='<Name>' metadata.version='<new>' | columns <expected_fields> | limit 10", hours: 1}
   -> keyed on the NEW metadata.version so you only see events the new parser touched
      (SDL never re-parses history)

A successful validation means: (a) sdl_put_file returned success, (b) the live query returns rows tagged with the new metadata.version, (c) the expected fields are populated and not null. If a subtype is rare and no event has arrived yet, widen the window or wait for one; do NOT fall back to synthetic ingest just to force a row.

If a field is missing, do NOT just retry, diagnose. Common causes: wrong escape level on a regex, a delimiter that didn't match (the format silently fails to apply), halt: true on an earlier format catching the line first, or discardAttributes dropping the field by mistake.

Version control & rollback (local project folder ONLY)

Keep every parser you touch under git in the local project folder, the user's Cowork project folder, and nowhere else. Do NOT commit parser definitions into the skill repo, an ai-siem checkout, the scratch/outputs dir, or any other location. The local project folder is the single source of truth for rollback.

On every deploy:

1. Mirror the parser to <project>/parsers/<name>.json in the local project folder.
2. On the FIRST touch of an existing live parser, commit the current live definition (from sdl_get_file) as the baseline BEFORE your edit; this is the rollback point.
3. After deploying, git add + git commit the new version there (commit message = the new metadata.version + what changed).

Rollback is then a local operation: git checkout <prior-commit> -- parsers/<name>.json, then sdl_put_file with path: "/logParsers/<name>", the previous content and expectedVersion: <current> to push it back. Bump metadata.version on the rollback too so you can confirm from the live stream which definition is active.

Fallback: synthetic HEC ingest (ONLY when the source is not live)

Use this only for a brand-new source that is not ingesting yet, or a shape you cannot observe in the live stream. Deploy to the real parser name (still bump metadata.version), ingest the sample through it, and query back. Do not resurrect the claude_test_* throwaway-parser pattern.

MCP tool calls (s1-secops-mcp), after deploying to the real parser path (see Default above):
 
1. hec_ingest     {logContent: <sample with a unique nonce>, parser: "<name>", endpoint: "raw"}
   -> authenticates with the SDL Log Write Key (S1_HEC_TOKEN); the key's own scope fixes
      where the events land, so no scope argument applies
2. Wait ~8 s (ingest-to-search latency).
3. powerquery_run {query: "parser='<name>' * contains '<nonce>' | columns <expected_fields>", hours: 1}
   -> confirm the expected fields are populated

Bundled references

When a question goes deeper than this file, read the relevant reference. Each is a focused deep-dive, load only what you need.

• references/parser-eligibility.md: Read this before anything else. Whether an SDL parser can be used on this source at all: the two attributes a parser can see (message, parser), the four cases that follow from that, PowerQuery probes for working out which case a source is in, and how to set the parser attribute from DPM when it is missing.
• references/ai-siem-catalog.md: Check this first before writing a new parser. Map of the public S1 parser repo (~150 parsers across community + marketplace), per-shape template recommendations, style-variance cheat sheet, and the fetch recipe.
• references/ocsf-mapping.md: Then start here for any new parser. OCSF class quick-pick (4001/4002/3002/1001/etc.), field-to-dotted-path mapping tables for Network Activity, Authentication, and File Activity, and the two authoring idioms: capture directly into dotted names vs capture vendor-native then rename via a mappings block. Also covers boolean/enum normalization (YES/NO/UNKNOWN → tls.is_encrypted). Points at ocsf-schema-documentation.md for authoritative field names.
• references/ocsf-schema-documentation.md: Authoritative OCSF field catalog, the full SentinelOne community-documented schema: 7 categories × 96 articles × ~25,759 field entries across every OCSF event class (System Activity, Findings, IAM, Network Activity, Discovery, Application Activity, Unified Alert Management). Grep this file for any OCSF field name you're about to emit; don't invent names. Always check this before choosing a dotted path, especially for classes beyond Network/Auth/File.
• references/syntax.md: Full augmented-JSON parser syntax: formats, patterns, attributes, lineGroupers, rewrites (including computeFields and timeDelta), special fields (timestamp/severity/message), discard, halt, repeat, association, intermittentTimestamps, skipNumericConversion. Read when authoring anything beyond a simple line format.
• references/parse-directives.md: Every {parse=...} sub-parser (json/dottedJson/escaped/urlEncoded/base64Encoded variants, strict variants for arrays, uri/uriMultivalue/uriAttributes, commaKeyValues, commaSeparated/pipeSeparated, sqlToSignature, syslogPriority, dateTime{Seconds,Ms,Ns}, hoursMinutesSeconds, seconds/milliseconds, bytes/kb/mb/gb, plus per-directive attrWhitelist/attrBlacklist rules). Read whenever the body of a field is itself structured.
• references/builtin-parsers.md: Catalog of all 16 built-in parsers (accessLog, cloudfront, json, dottedJson, dottedEscapedJson, elb-access, heroku-logplex, keyValue, leveldbLog, mysqlGeneralQueryLog, mysqlSlowQueryLog, postgresLog, redshift, s3_bucket_access, spot_instance_data, systemLog) and when to alias vs override. Read first when sizing up a new log source; you may not need to write a parser at all.
• references/mappers.md: mappings block (gron-style transformations: cast, copy, copy_tree, drop, drop_tree, hash, reduce_array, rename, rename_tree, replace, zip), array index syntax, predicate semantics. Read when restructuring events to OCSF or another target schema.
• references/testing-workflow.md: Detailed validation recipe with the sdl-api skill, including how to scope queries with a unique host tag, how to interpret common error responses, and how to clean up.
• references/per-app-sentinel.md: The per-app sentinel pattern for a single parser that fans many services or applications out to their own OCSF classes: pattern overview, the add-a-service checklist, and the periodic catch-all audit query.
• references/onboarding-learnings.md: Tenant-validated learnings from onboarding new sources via the sdl-solutions playbook (dotted-prefix JSON capture, mappings version requirement, account-level parser scope, activation latency, sourcetype binding, IP-keyed network-source enrichment).

Bundled scripts

• scripts/lint_parser.py: offline parser lint (workflow step 4a). --new for a parser you are authoring, --json / --csv for a report over a directory, --rules to list the checks.
• scripts/parser_rules.py: the rules themselves and a tolerant augmented-JSON reader. Single source of truth: the eval grader's lint_clean assertion imports it. Add a rule here only for a trap this file already records as tenant-validated, and give it a known-bad and known-good case in tests/test_lint_parser.py.

Bundled examples

Annotated, runnable parser definitions. Copy and adapt rather than starting from scratch.

• examples/01-cef-over-syslog.json: Syslog-framed CEF (<PRI>timestamp host CEF:0|...|...|key=value). Demonstrates: syslogPriority parse, named timestamp pattern, positional pipe-delimited CEF header, key/value catch-all on the extension.
• examples/02-json-with-envelope.json: <timestamp> <host> {"...json..."}. Demonstrates: timestamp + host extraction, {parse=dottedJson} on the body, discardAttributes: ["message"].
• examples/03-key-value.json: Pure key=value lines. Demonstrates: the $_=identifier$=$_=quoteOrSpace$ repeating idiom and a leading static prefix.
• examples/04-multiline-stack.json: Java/Python stack traces. Demonstrates: lineGroupers with continueThrough, attribute tagging.
• examples/05-rewrite-and-mask.json: Single line format plus rewrites to mask password=... values and compute a derived field via computeFields.
• examples/06-alias.json: One-line alias parser ({ aliasTo: "json" }) for the case where a built-in already does the job.
• examples/07-juniper-srx-rtflow-ocsf.json: Juniper SRX RT_FLOW_SESSION_CREATE (structured-data syslog, RFC 5424-ish). Demonstrates: the vendor-native capture + `mappings.rename` to OCSF idiom end-to-end (source-address → src_endpoint.ip, nat-source-address → connection_info.src_translated.ip, etc.), repeating key/value catch-all on the structured-data body, computeFields for protocol-id → connection_info.protocol_name and encrypted YES/NO → tls.is_encrypted, Network Activity 4001 class tagging.
• examples/08-gron-capture-template.json: The PARSER_TEMPLATE shape lifted from ai-siem: capture the entire event with $unmapped.{parse=gron}$, then do every rename/copy/cast/constant in a single v1 mappings block. Ideal scaffold for JSON-shaped sources when you want all OCSF work in one place.
• examples/12-linux-auditd-ocsf.json: Linux auditd over syslog (<pri> ts audispd: type=<REC> msg=audit(<epoch>:<id>): k=v ...). Demonstrates: a type=-anchored record-type + audit-event-id capture (with the leading space that avoids the nametype= collision), the KV catch-all, v1 first-match-wins mappings fanning the record type out to OCSF classes (SYSCALL/EXECVE/PROCTITLE → Process 1007, PATH/CWD → File 1001, USER_/CRED_ → Authentication 3002, ANOM_*/AVC/SECCOMP → Detection Finding 2004, SOCKADDR → Network 4001), per-op predicate status/severity derivation, and cleaning the EXECVE arg vector into a readable process.cmd_line via a computeFields PowerQuery replace() (the replace mapper op is a no-op and there is no hex decode for proctitle).
• examples/README.md: When to pick which template.

Summary of the contract you're holding

When a user pastes a log and asks you to parse it, you owe them: (1) a parser file written against the SDL augmented-JSON DSL, (2) proof it actually extracts the right fields when ingested through the live SDL pipeline (not just the editor preview), (3) a clean handoff (final path, fields, sample query). The bundled references and examples exist so you don't have to hold the entire DSL in your head, load them as needed and reach for the templates first.

Parser deployment via s1-secops-mcp

Parser deployment and validation use the s1-secops-mcp MCP tools, which run on the user's machine and reach *.sentinelone.net where the Cowork sandbox cannot. Use sdl_put_file, sdl_get_file, sdl_list_files, hec_ingest and powerquery_run directly; the sdl-api skill's Python scripts are host-only. If a tool reports a missing credential, ask the user to run s1-secops-mcp setup on their machine.

Per-app sentinel pattern (multi-tenant / multi-service parsers)

When one parser handles events from many distinct services or applications, each needing its own OCSF class assignment, use the per-app sentinel pattern: extract a discriminator field, add one format-id sentinel per service, list every sentinel field in discardAttributes, add one v1 first-match-wins mappings block per sentinel (drops duplicated into each block because v1 is first-match-wins), and end with a predicate: "true" catch-all placed last. The full pattern overview, the how-to-add-a-new-service checklist, and the periodic audit query for services stuck in the catch-all are in references/per-app-sentinel.md.

Onboarding learnings (tenant-validated 2026-06-13, <console>)

Tenant-validated learnings from onboarding new sources through the sdl-solutions playbook, JSON-per-line dotted-prefix capture ($unmapped.=json{parse=dottedJson}$), the mappings version: 1 plus transformations requirement, account-level parser scope, 3 to 5 minute activation latency per deploy, sourcetype-to-parser binding, and IP-keyed network-source enrichment, are in references/onboarding-learnings.md.