kid-sid/claude-spellbook/deployment-strategies

Deployment Strategies

A reference for selecting and implementing deployment strategies that minimize risk, enable zero-downtime releases, and provide fast rollback paths.

When to Activate

• Planning a deployment strategy for a new service or major release
• Implementing feature flags in an application
• Coordinating a database migration with a zero-downtime deployment
• Setting up canary releases or progressive delivery
• Defining rollback procedures for a service
• Reducing deployment risk for a high-traffic service

Strategy Comparison

Rolling Updates (Kubernetes)

Default Kubernetes behavior when you run kubectl apply. Pods are replaced incrementally — no full restart required.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # max pods above desired count during rollout
    maxUnavailable: 0  # never go below desired count (zero-downtime)

• Set maxUnavailable: 0 to guarantee zero downtime — new pods must pass readiness probes before old pods are terminated.
• Rollback: kubectl rollout undo deployment/my-service
• Target a specific revision: kubectl rollout undo deployment/my-service --to-revision=3
• Monitor progress: kubectl rollout status deployment/my-service
• Issue: slow rollback if many replicas; new version runs alongside old — both app versions must be compatible with current DB schema.

Blue/Green Deployments

Two identical environments run in parallel: Blue (live) and Green (new version). Traffic flips atomically from one to the other.

Process

1. Deploy new version to Green environment
2. Run smoke tests against Green (no user traffic yet)
3. Flip traffic: update load balancer rule or Kubernetes Service selector
4. Monitor error rate and latency for 15–30 minutes
5. Decommission Blue (or keep as instant rollback for 24 hours)

Kubernetes Implementation

Flip the Service selector to switch which deployment receives traffic.

# Blue deployment (live)
spec:
  selector:
    app: payment-service
    version: blue   # Service points here
 
# Green deployment (new)
spec:
  selector:
    app: payment-service
    version: green  # Update Service to point here after smoke tests

Flip command:

kubectl patch service payment-service -p '{"spec":{"selector":{"version":"green"}}}'

Considerations

• Cost: 2x infrastructure during transition window.
• Warm-up: Green must receive warming traffic (health checks, cache pre-warming) before the flip to avoid cold-start latency spikes.
• Database: Both Blue and Green versions must be compatible with the same DB schema during the transition window. Use the expand-contract pattern for migrations.

Canary Releases

Gradually shift traffic from the stable version to the new version. Automated analysis gates promotion based on SLO metrics.

• Typical progression: 5% → 25% → 50% → 100%
• Automated promotion: if error rate < 1% and p99 latency < 500 ms, advance
• Manual gate: require human approval before advancing beyond 25%
• Automated abort: if metrics breach thresholds, roll back instantly

Argo Rollouts

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payment-service
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: {}  # manual gate — requires human approval
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
      analysis:
        templates:
          - templateName: error-rate-check
        startingStep: 1
        args:
          - name: service-name
            value: payment-service

Promote or abort the rollout:

kubectl argo rollouts promote payment-service  # advance to next step
kubectl argo rollouts abort payment-service    # rollback to stable

Flagger (Linkerd / Istio)

Flagger integrates with service meshes for automatic traffic splitting and metric-based promotion. Define a Canary CR with analysis.metrics referencing Prometheus queries. Flagger handles weight increments and rollback automatically — no manual step definitions required.

Feature Flags

Why Feature Flags

• Decouple deployment from release: deploy code, enable for users later
• Progressive rollout: enable for 1% → 10% → 100% of users without redeploying
• Kill switch: disable instantly without a deployment or rollback
• A/B testing: different experiences for user segments based on targeting rules

Flag Lifecycle

1. Add flag (disabled by default)
2. Deploy code wrapped behind flag
3. Enable for internal users → beta users → percentage rollout → 100%
4. Remove flag and dead code (flags are technical debt — clean up within a sprint of full rollout)

Tools Comparison

Code Pattern (OpenFeature)

import { OpenFeature } from '@openfeature/server-sdk';
const client = OpenFeature.getClient();
// Simple boolean flag
const isNewCheckoutEnabled = await client.getBooleanValue(
  'new-checkout-flow',
  false,  // default value — returned if flag is missing or evaluation fails
  { targetingKey: userId }
);
if (isNewCheckoutEnabled) {
  return newCheckoutHandler(req, res);
} else {
  return legacyCheckoutHandler(req, res);
}

OpenFeature's provider abstraction means swapping from LaunchDarkly to Unleash requires changing only the registered provider — application code stays the same.

Database Migrations and Zero-Downtime Deployments

The Problem

Direct ALTER TABLE can lock tables under load. Renaming columns breaks the old app version that runs alongside the new version during a rolling deploy. Any migration that removes or renames a column must be done in phases.

Expand-Contract Pattern (Parallel Change)

Use for: adding NOT NULL columns, renaming columns or tables, changing data types.

Phase 1 — Expand (additive only):

• Add new column as NULLABLE
• Deploy application code that writes to both old and new columns
• No downtime — old app version still works with the old column

Phase 2 — Migrate:

• Backfill existing rows in batches to avoid table locks:

``sql UPDATE table SET new_col = old_col WHERE new_col IS NULL LIMIT 10000; ``

• Deploy application code that reads from the new column
• Add NOT NULL constraint once all rows are populated (now safe)

Phase 3 — Contract (remove old):

• Deploy application code that no longer references the old column
• Drop old column in a separate migration
• Can be done in a later sprint once confidence is high

Example Timeline

Renaming user.username to user.display_name:

Sprint 1: Add display_name (nullable), write to both columns
Sprint 2: Backfill rows, read from display_name, add NOT NULL
Sprint 3: Remove username column

Large Table Migrations

For tables with millions of rows, use pt-online-schema-change (Percona) or gh-ost (GitHub) to perform the migration on a shadow table and cut over with minimal locking.

Rollback Procedures

When to Roll Back

Roll back when:

• Error rate exceeds SLO threshold (e.g., > 1% errors) within 15 minutes of deploy
• p99 latency increases more than 2x baseline
• Critical functionality is broken (payments, login, data integrity)

Do not roll back immediately for:

• Cosmetic issues or minor UI regressions
• Minor performance variance within acceptable range
• Cases where rollback itself would cause different data loss (evaluate carefully)

Rollback Decision Tree

Error rate > SLO?
├── Yes → Can we fix forward in < 15 minutes? → No  → ROLLBACK
│                                              → Yes → hotfix + monitor
└── No  → Monitor, do not rollback

Rollback Commands

# Kubernetes rolling update — undo last rollout
kubectl rollout undo deployment/payment-service
 
# Kubernetes — target a specific revision
kubectl rollout undo deployment/payment-service --to-revision=3
 
# Argo Rollouts canary — abort and revert to stable
kubectl argo rollouts abort payment-service
 
# Helm — rollback to a previous release number
helm rollback payment-service 3

Rollback Runbook Template

## Rollback: [Service Name]
 
**Trigger criteria:** [e.g., error rate > 1% for 5 minutes]
 
**Steps:**
1.Notify on-call channel: "@oncall rolling back payment-service due to [reason]"
2.Run: `kubectl rollout undo deployment/payment-service -n production`
3.Verify: `kubectl rollout status deployment/payment-service`
4.Check metrics: confirm error rate returns to baseline
5.Create incident ticket with timeline and root cause
 
**Data rollback:** [specify if DB migration rollback is needed and how]
**Escalation:** [who to page if rollback fails]

See also: ci-cd, containerization, observability, incident-response

Red Flags

• Deploying a schema migration and an app change in the same atomic release — if the migration succeeds but the app rollout fails mid-way, old pods still running see the new schema; migrations and app deploys must be sequenced across separate releases
• Setting `maxUnavailable: 1` instead of `0` for critical services — during a rolling deploy, one pod is taken down before the new one is ready, briefly dropping capacity below the desired replica count and increasing error rates
• Feature flag with no documented cleanup date — flags that ship but never get cleaned up accumulate into untested conditional branches; enforce a sprint deadline at the time of flag creation
• Blue/green flip without traffic warming on the Green environment — an un-warmed JVM or cold connection pool on Green produces a latency spike immediately after the flip that looks like an outage
• Canary rollback based only on error rate, ignoring latency SLO — a new version can stay under 1% errors while p99 latency doubles; always gate canary promotion on both error rate and latency thresholds
• Defining rollback criteria only after an incident starts — ad-hoc rollback decisions under pressure are slow and inconsistent; criteria and commands must be written in the runbook before the deploy
• Rolling back a migration by dropping a column that the old app version still reads — the old app immediately errors after the column is dropped; contract phases must be fully completed before any column is removed
• Using environment variables as a feature flag substitute for runtime toggles — env var flags require a pod restart to take effect and cannot be changed per-user or per-percentage; use a proper feature flag service for runtime control

Checklist