latent-sre/save-toolkit/resilience-analysis

Resilience analysis

Produce a supported engineering decision about a service's failure behavior. Load stack-profile for applicable ownership and platform constraints. This method grants no access, execution, design approval, or change authority. Use available source and observations; an experiment remains proposed until an authorized actor supplies results from an established verification environment.

Establish the outcome and evidence

Name the critical user transaction or batch/data outcome, environment, relevant requirements and scope. Use accepted SLOs, recovery objectives and constraints; missing targets remain unknown or explicit proposals for the human owner. External practices are guidance, not company policy.

Compare intended behavior (design/runbook), implemented behavior (effective code/configuration and tests), and observed behavior (targeted telemetry or exercise results). Bind each to its revision, deployment and time. Conflicting or stale records are leads to reconcile, not permission to choose the convenient one. Source-only work can establish a conditional risk, not current production impact.

Trace and challenge a failure path

1. Follow the user and data paths, including asynchronous work, shared resources and external

dependencies. Inspect actual callers and configuration overrides, not just definitions.

1. Select credible failure or saturation conditions. For the relevant workload read

failure scenarios; do not apply every scenario to every service.

1. Trace trigger -> component behavior -> propagation -> user effect -> protection -> recovery.

Record the existing controls and seek counter-evidence: overall deadlines, cancellation, admission limits, replay/deduplication, failover capacity, or a deliberately accepted degraded mode.

1. Choose the smallest observation or controlled experiment that distinguishes the remaining

explanations. An inaccessible source or failed query leaves a gap. Do not claim an alternative ruled out because it was not observed, nor trigger a production fault to demonstrate a risk.

1. Assess normal, degraded, overloaded and recovering behavior where applicable. A successful

restart, cleared alert, completed backup or reverted commit does not by itself prove user recovery or data integrity. Use database-reliability for data and restore mechanics and root-cause for diagnosing an observed failure.

Substantiate and prioritize

A material finding needs the outcome/requirement, reachable trigger and mechanism, evidence, controls checked, consequence and a useful next action. Classify the result separately from its evidence label:

Use [verified], [sourced], [unverified] with their target, method and time; preserve [UNTRUSTED] data through summaries. Inspect beyond missing pattern names: no circuit breaker is not a defect when other controls meet the requirement. No observed incident is not proof of resilience. Allow no material finding within the inspected scope and retain coverage gaps.

Prioritize user impact, plausible exposure, recovery difficulty, uncertainty and engineering cost. State why the proposed work outranks alternatives; avoid invented probability scores. A missing required verification can matter urgently without proving that recovery will fail.

Engineer the improvement and proof

Compare the smallest effective change with existing controls and consequential alternatives. Specify the mechanism it changes, compatibility, degraded behavior, operational burden, recovery, owner and acceptance criteria. Load eng-ladder for consequential shared-contract decisions. Implementation, independent change review and live execution retain their existing owners.

For calculations, show units, measured versus assumed inputs, the equation, uncertainty and validity conditions. Account for shared demand and failure capacity. Use existing permitted calculators when available; no tool capability may be invented. A model estimate is not measured load-test evidence. For retry timing, distinguish retries from total attempts and requested timeouts from enforced durations. Trace false returns, exceptions, backoff and cancellation before claiming a time bound; a timeout sum alone bounds neither elapsed time nor slot occupancy. A concurrency limit protects shared capacity only if admission or waiting does not retain the slots being reserved.

Verification names the target/version, stimulus, expected user and resource behavior, discriminating measurements, bounds/stop conditions, and recovery checks. To verify an admission or isolation control, drive arrivals above completion capacity until its limit and overflow path are exercised, while protected work runs concurrently. Count active work, permit waiters and retries in shared-slot occupancy; representative demand alone may never exercise the protection. Preserve actual results separately from the proposed test. Assess findings and coverage, not a blanket reliability score; test passes do not approve a deployment. See worked contrasts for supported risks, effective controls and missing evidence. Read sources when checking a practice's provenance or adapting this method.