rbadillap/sre/reversible-first

Reversible first

Mitigate first with what can be undone, leaving anything irreversible to a person.

When

Whenever users are affected and a choice of action exists, and whenever a skill is about to propose a change in production.

When not

When the mitigation itself would destroy what is needed later: evidence in a security incident, or data that a restore would overwrite. Then preserve first, and mitigate in a way that keeps it.

How

1. Prefer the action that can be undone:

1. Check the second-order effect before acting. Ask what the action does to everything around it. Scaling an application up can make a saturated database worse, because each new instance opens more connections. Restarting pods can drop work in progress.
2. Write how to undo it, before doing it. If the undo can't be written, the action isn't reversible.
3. Hand irreversible actions to a person. Deleting, restoring, failing over, migrating data and changing access always end in a proposal: what, why, the risk, and how it would be undone if it could.

• Handing it to a person doesn't replace checking the target. The proposal names the environment, the resource and its current role (primary or replica, for example), from a recent check, with the scope and when to stop. A more visible prompt helps, but it doesn't prove the target is right.
• Ask for an independent review when the risk or the team's policy calls for one.
• If the person can't confirm the target, or can't go on safely, that step stops, and they ask for support or hand off.

1. Confirm it worked. After a mitigation, check the user-facing signal recovered. If it didn't, undo it and reconsider the cause.

The mistake it prevents

Either users wait while the team searches for the root cause, or a fast fix causes a second incident.