version: "1.0.0" name: high-availability description: Preflight, enable, and operate Gateway Docker Workload Availability, which places a Container, blue/green Deployment, or whole Compose Project on several independent Docker Nodes in replicated or failover mode. Use when asked to make a workload highly available, add replicas, set up failover, check an availability policy, or recover after losing a Node. Business and Enterprise only. Use after using-gateway, and after deploying-workloads for the underlying workload.

High Availability

Docker Workload Availability adds multi-node placements to an existing standalone Container, blue/green Deployment, or whole Compose Project. It creates no new resource type: a policy attaches to the workload and from then on owns placement count, generations, routing membership, and cleanup. It is a Business and Enterprise feature.

Start with using-gateway for connection, discovery, and safety rules, and deploying-workloads for the workload itself. One tool covers the lifecycle: manage_docker_availability({ operation, ... }), gated by docker:availability:manage (folder grants resolve through the workload's folder; the scope also implies docker:containers:view).

Modes

Replica count is manual. There is no metric autoscaling and never more than one placement of a workload per Node.

Priority mode

Either mode can prefer Nodes in a fixed order: priorityMode: true with nodePriority, an ordered list of Node IDs whose first entry is the primary and the rest backups in order. It may only contain eligible Nodes (the selectedNodeIds with selected mode) and no duplicates; selectedNodeIds still decides eligibility. Serving placements go to the first available Nodes of the order; Nodes outside it come after. When a higher-priority Node returns and stays healthy for failbackDelaySeconds (default 300, 0 to 3600), a failback operation starts the workload there, switches traffic to it, and then drains and removes the backup placement, so a placement serves throughout (this needs rolloutPolicy.maxSurge of at least 1, or more than one replica and maxUnavailable of at least 1). In lease mode a failback is a handoff with a short gap instead; see Data-plane failover. A Node that goes offline or reports an error again restarts the delay, so a flapping Node does not take traffic back; after a Gateway restart every Node's delay starts again. Failback runs only while the policy is healthy and no other operation of the same policy is active; operations of different policies run in parallel (see Operations). preflight warns AVAILABILITY_FAILBACK_NEEDS_SURGE when the rollout policy cannot move without a gap, and then no failback runs. A failed failback leaves the backup serving and is not retried automatically for 15 minutes; retry_operation retries it sooner. get returns priorityMode, nodePriority, and failbackDelaySeconds; failbacks appear in list_operations. With priorityMode: false (the default) placement follows free capacity as before.

Prerequisites

• At least two online, compatible Docker Nodes with capacity for the replicas plus temporary placements during rollout.
• No configured or observed mounts of any kind: named, external, read-only, or host bind. For Compose, the whole project is checked. Availability never copies local data between Nodes, so persistent state must live outside the workload, for example behind a managed database binding (databases).
• Images are pinned by repository and digest in the internal registry and pre-pulled on standby Nodes. Verify registry access, Relay and Secure Link paths, and dependent managed databases beforehand.

Workflow

1. preflight with resource: { type: "container" | "deployment" | "compose", nodeId, containerName? | deploymentId? | composeProjectId? }, mode, and optional desiredReplicaCount, nodeSelectionMode, selectedNodeIds. It reports eligibility and candidate Nodes. Resolve every incompatibility (mounts, capacity, Node compatibility) before continuing.
2. enable with the same shape, plus optional rolloutPolicy: { maxUnavailable, maxSurge, drainSeconds }, offlineReplacementGraceSeconds (how long to wait after losing a Node's control connection before creating a replacement; about 15 seconds by default), and partitionMode (strict, the default, or available; see Data-plane failover).

Enabling a workload that already serves never interrupts it and never starts a second copy beside it: the running Container, Deployment, or Compose Project becomes the first placement where it runs, whatever nodePriority says (a failback moves it afterwards). A Container is never recreated; a Deployment or Compose Project that runs the mirrored image is adopted as it runs. One that cannot be adopted (it needs the managed-database projection or runs another image) is replaced without a gap where two copies are allowed (replicated, or failover with partitionMode: available), otherwise with a short planned handoff. Routes keep reaching the origin through their own Secure Link until a placement serves through its member link; the switch then keeps the old link 5 more seconds for nginx's old workers.

1. Poll get or get_by_resource, and list_operations with policyId, until the requested serving count is reached.
2. Verify real traffic through the Route, database access from the placements, and application logs, not only the placement count.
3. update with policyId changes mode, replica count, Node selection, rollout policy, grace period, partitionMode, or priority mode (priorityMode, nodePriority, failbackDelaySeconds). retry_operation with operationId retries a stuck rollout.
4. disable needs policyId, the survivingPlacementId to keep, and confirmation set to the exact typed text the tool or Console shows. It is destructive; never guess the confirmation text or the surviving placement. The survivor keeps running and becomes the plain workload as it runs, with its own restart policy back: an origin Container, Compose Project, or Deployment is adopted in place, and so is a Deployment copy on another Node; a Container or Compose copy on another Node (it runs under a name Availability generated) is replaced by a copy with the original name on that Node, started before the old one is removed, so traffic has no gap. An adopted origin Compose Project shows as drifted until the next apply. In lease mode the disable first closes the lease with the survivor kept running. While it runs, no heal, scale, failback, or start of the policy runs (AVAILABILITY_DISABLE_IN_PROGRESS), a waiting disable keeps the status disabling, and a retry keeps the survivor chosen first; the disable removes the other placements itself.

Data-plane failover (lease mode)

A policy runs in lease mode when every ingress nginx Node of the workload's routes and every relay that carries it advertise availability_lease_v2, and enough members that can vote exist for a quorum. A policy on the backend path enters lease mode only once every candidate Node, ingress nginx Node, carrying relay and witness has run availability_lease_v2 (with its lease watchdog and identity) for 2 minutes without a restart; until then reason is participants_settling with the Nodes or relays still settling, so a fleet in the middle of an update (Gateway first, then the Nodes one by one) never switches. In lease mode failover no longer depends on Gateway being reachable.

• Who decides. The Nodes and relays hold a lease per serving slot. A serving Node renews it every few seconds and stops its own copy if it cannot, so a dead or cut-off Node is replaced by the next candidate within about 45 seconds even while Gateway is down.
• Standbys. They are created ahead of time: the image is pulled and the container is created but not started. Preparing a standby again (heal, reconnect, maintenance) changes nothing unless its spec, its image (by image ID), or its managed-database projection changed; a standby whose projection changed (for example a rotated binding credential) counts as not ready and is prepared again. After the upgrade to 2.11 every standby with a database binding is prepared once more, because the projection it was prepared with is recorded from then on.
• Traffic. It reaches only the current lease holder. Every placement's member Secure Link is registered on every relay that advertises availability_lease_v2; standbys stay registered but dormant, so a new holder is reachable through every relay at once. A handoff flips the members in place (old holder dormant, successor serving) without creating a link or relay endpoint.
• Release. A holder that stopped its copy for a local reason (lease timer, lost watchdog, detected freeze, closed lease) releases the slot once the stop is confirmed, so the next candidate starts within seconds instead of after the ~33-second hold.
• Lease watchdog. gateway-lease-watchdog runs on every Docker Node that can hold a slot, installed by the node installer or by the daemon itself when it runs as root, and updates itself. It kills a lease-mode container past its deadline even when the daemon or dockerd hangs. A watchdog stalled for up to ~8 seconds stops nothing; one missing for ~10 seconds makes the holder stop its copy and release the slot.
• Freezes and clocks. Lease timers use the monotonic clock only; an NTP step of any size or direction never stops a copy. A holder whose host or VM was paused for more than ~2 seconds notices it from the clocks of the first relay or voter it hears after resuming and stops its copy at once (fence reason host_frozen in the daemon log); a lease acquired after the freeze is never fenced for it. A hung dockerd on the holder does not stop traffic to the running copy.
• One slot per Node. A Node never holds two slots of one policy and takes its own last slot back first.
• Voters. Each policy has its own voters: its candidate Nodes, one per physical host, in takeover order, plus witnesses so the count is odd and at least 3 (at most 7). Only Nodes and relays that advertise availability_lease_v2 and reported a lease identity vote. A witness is a relay or Docker Node on another host than every candidate: the one set in witness, else one picked automatically, which is never Gateway's local relay while another ready relay can witness (the local relay stops with Gateway). An automatic witness stays until it can no longer vote, so round-trip jitter never changes the voters. A three-candidate policy needs no witness. One live relay is enough as long as a majority of the voters is reachable through it.

Entering lease mode never interrupts a serving copy: only a lease formed after the switch satisfies a slot's bootstrap, so an old commit from an earlier lease period never makes a Node stop its own copy, and the serving copies' links change role in place. Lease entry also waits for a running enable or rollout of the policy.

Excluded Nodes. A condition of one Node never changes the policy's mode. A candidate that is offline, whose lease watchdog is not running (watchdog_missing), whose daemon does not advertise availability_lease_v2 (daemon_outdated), or that has not reported a lease identity yet (identity_pending) is listed in lease.excludedNodes with that reason. It gets no new standby, no failback or other handoff moves to it, and its daemon does not take a slot; the next candidate does. A Node holding a slot is never cut off by Gateway for this: a Node without a watchdog stops its own copy, and an outdated holder keeps its slot until it is updated or hands over. An outdated or unidentified Node leaves the voters and the candidate list only after the condition lasted 2 minutes, so a daemon restart or a rolling update changes nothing. Fix the Node (start the watchdog or re-run the node installer, update the daemon) and it takes part again.

Leaving lease mode. Only when lease mode is impossible: fewer members that can vote than a quorum (insufficient_voters), an ingress nginx Node or a carrying relay without availability_lease_v2 (ingress_not_capable, relays_not_capable), no candidate that can hold while no slot is held, a Gateway edition without data-plane failover (controller_unsupported, a Community build), or an explicit disable or lifecycle operation (stop, start, restart). The license never does: per the license contract nothing stops after a license loss, so no license state change closes lease mode. Apart from the explicit requests, the condition must last 2 minutes without a break; until then the policy stays in lease mode and lease.reason.since tells since when it has been impossible.

Leaving never stops the serving copy. The policy passes through closing: Gateway publishes a closed lease that names, per slot, the Node holding it as the retained holder (lease.retainedHolders). Once a majority of the voters confirm the close to that Node it keeps its copy running without a lease (its watchdog no longer guards that copy) and the backend path adopts it as running, usually within seconds; nothing restarts it. A holder that cannot reach a majority (cut off during the close) stops its copy as any lease holder does, and the backend starts that slot again once its lease has expired (about 47 seconds after a voter majority saw the close). A close keeps every placement, the origin included: standbys stay prepared and nothing is queued for cleanup, the backend path starts a prepared standby before it creates a new placement, and a later re-entry takes the kept standbys as they are. Only a stop of the workload stops the copy, after the close; a disable keeps the surviving placement running and removes the others itself.

What Gateway still does in lease mode:

• plans moves: failback, drain, manual moves, rollouts;
• keeps two standbys provisioned and heals: a policy short of serving copies or standbys with eligible online capacity gets maintenance every 15 seconds and on every Node connect; a returning Node gets its free slot back (its placement is kept, a missing one is recreated), a Node holding two slots hands one to a standby elsewhere, and a returning Node's placements take their state from its daemon (a stopped copy becomes a ready standby, so failback runs after partitions as after outages); a busy daemon delays this with the retryable AVAILABILITY_DAEMON_BUSY instead of failing placements;
• keeps serving paths through its own return: a member link whose refresh fails because a Node has not reconnected keeps its relay endpoint and is retried, and a relay that reconnects gets every policy change at once;
• reconciles its records with the actual holders after it returns, within seconds of starting: the local relay reports the holder before the Nodes reconnect. An autonomous takeover appears in the audit log as docker.availability.lease_failover, a planned move as docker.availability.lease_handoff, and a slot that lapsed and was taken again by the same Node (it was stopped or cut off, also while the Gateway was down) as docker.availability.lease_reacquired. Each is dated at the time the holder acquired the slot (its own report, corrected by its measured clock offset, else the earliest voter sighting, else when Gateway noticed); a daemon restart whose copy kept its lease is no new holding and changes nothing. Without the holder's own time the row is written once the time settles: when the holder reports, else after 60 seconds with the voters' time, else after 5 minutes. The waiting entry is stored in the database, so a Gateway crash loses nothing and never writes it twice. Details carry takeoverAt, noticedAt, takeoverSource (holder, voters, noticed) and takeoverId; holderSince is corrected as better reports arrive.

A planned move of a slot (failback, drain, manual move, nodePriority change) is a handoff: the serving Node stops its copy and releases the lease, and only then does the next Node start its own. The slot serves nothing for the old copy's stop plus the new copy's start and readiness (a container without a health check counts as ready after 3 seconds), usually 5 to 15 seconds. A strict policy never runs two copies of a slot, so in failover mode, which has one slot and no surge, requests fail for that time; in replicated mode the other replicas keep serving. Schedule failbackDelaySeconds and priority changes accordingly.

get returns a lease object:

• mode: legacy, bootstrapping, lease or closing;
• reason: why a policy is legacy (for example participants_settling while the fleet settles after an update), or why lease mode is impossible right now, with since while a lease-mode policy waits out the 2 minutes. nodeIds or relayIds name what to fix. With watchdog_missing, the listed Nodes have no running lease watchdog; when their daemon cannot install it (it runs without root), re-run the node installer there;
• excludedNodes: [{ nodeId, reason }] as above;
• retainedHolders while closing: [{ slot, holderNodeId, confirmed }], the copies that keep running through the close (confirmed once a voter majority confirmed it);
• holders per slot, with Node, placement and holder time;
• voters and witness (with warning: witness_near_candidate, no_eligible_witness or configured_witness_unavailable);
• voterMargin with voters, reachable, required and margin. It counts the voters that can vote without Gateway: Gateway's local relay does not count, and a Node counts only while it keeps a lease connection to another relay. Every Node of a lease-mode policy connects to every relay for that.

When margin is 0 or less, losing one more voter disables autonomous failover. A two-candidate policy with a remote relay has margin 1.

Updates. Gateway sequences daemon and relay updates of lease participants itself: an update of a Node or relay that votes in or can hold a lease-mode policy waits until the policy's other voters and candidates are back online, reported their lease state and vote again (get_node shows metadata.updatePhase: waiting_for_lease_peers and metadata.updateWaitingFor; the update request returns leaseSequenced: true). Requests sent together run standbys first, then holders; Nodes sharing no policy update in parallel; a peer unsettled 3 minutes after its restart stops blocking and a request that waited 30 minutes fails naming the peers. So several HA Nodes can be requested at once. The wait matters mostly for the first restart after upgrading from a release before 2.11, and after a reboot or with fresh state, which abstain from voting for about 33 seconds. A holder whose daemon restarts on the same boot keeps its slot and keeps serving: it takes traffic again as soon as the new process registers. A graceful daemon restart is announced to the relays, which hold its registrations up to 15 seconds, and ingress Nodes hold new connections up to 8 seconds instead of failing them; a replica with another serving replica fails over at once.

partitionMode controls a network split:

• strict, the default: never runs two copies.
• available: keeps serving on both sides of a split and may briefly run two copies. Never set available for singletons such as indexers, queue consumers or cron jobs.

Operations

Each policy runs one operation at a time, in order (list_operations). Operations of different policies run in parallel: background operations (heal, scale, rollout, enable, cleanup) for at most 4 policies at once, while failback, start, stop, restart, and disable never wait for that limit. A returning Node therefore fails back several workloads without queueing behind other policies' heals.

Stop and start

stop (through the workload's lifecycle tools) stops the copies that run and keeps every placement: standbys stay prepared and the policy stays healthy while stopped. start starts the copies that served last, never a standby, and keeps the standbys. Neither removes, recreates or fails a placement, and the origin is never deleted (not by closing a lease, stale cleanup, or as a surplus standby; on a deselected Node it stays, stopped). In lease mode the operation closes the lease first; the close itself stops nothing.

Node loss and recovery

In lease mode, see Data-plane failover. Otherwise, after a Node's control connection is lost (shown offline after 5 seconds; its placements leave routing after 10), Gateway waits offlineReplacementGraceSeconds and then starts a prepared standby, or creates a replacement on an eligible Node when there is none. In both modes a policy below its serving count with eligible online capacity is healed without an edit (checked every 15 seconds outside a Node's offline grace and on every connect), and an update that sets the current count or Node selection again repairs missing placements. When the lost Node reconnects, Gateway heals the policy and excludes stale generations from routing; it does not fence the host or guarantee single-writer semantics outside its own routing. Never delete child Containers to force an operation forward; that desynchronizes the durable record. If the replica count is not restored, check operation phase, Node compatibility and capacity, image pull, application health, and private dependencies, in that order.

A rolling update that fails repeatedly (3 attempts or 15 minutes) rolls back automatically; updated replicas return to the previous image and settings.

Routing and bindings

Routes, Additional Routes, and Advanced Secure Links target the logical workload. Gateway projects healthy placement endpoints and balances new connections by least connections; existing connections do not move. A placement takes traffic only once it is ready: its image health check is healthy, else its port accepts connections; for a Deployment the active slot's app must answer through the router (router-generated errors carry X-Gateway-Deployment-Router: upstream-unavailable). A placement that does not serve keeps its ingress socket closed in every mode, so nginx fails the connect before sending anything. nginx retries the next placement on connect errors, and on 502/503/504 for idempotent requests (at most 5 tries in 10 seconds; POST, PATCH and LOCK are never re-sent once they reached a placement); a failed placement is skipped for one second, and every placement is also listed as a backup, so a Route never has "no live upstreams" while one serves. The relays drop a placement whose host stops answering within about 3.5 seconds; before that a new connection to it costs at most 2 seconds. Route health is checked through the member links. Managed database bindings get per-placement connections tied to the logical resource. Use the placement selector in logs, console, and monitoring to diagnose one instance.

Verify

After enable, get shows the desired serving count with every placement healthy, a Route request succeeds, and database traffic (when bound) arrives from more than one placement. In lease mode, lease.mode is lease with a holder per slot, excludedNodes is empty and voterMargin.margin is at least 1. After a real or simulated Node loss, a replacement appeared (lease mode: within about 45 seconds; otherwise within the grace window) and routing excluded the stale placement.

Pitfalls

• Availability protects the workload, not Gateway itself, nginx, the database engine, registry storage, or shared volumes. Check each dependency's own resilience.
• Any mount, even a read-only one, fails eligibility. There is no partial mode that skips volume checks.
• Compose Availability replicates the whole project per placement; it does not spread individual services across Nodes.

Further reading: Availability, Data-plane failover, Compatibility and limits.