Skill v1.0.0
currentTrusted Publisher100/100name: substrate description: > How AX depends on Agent Substrate (modules, address contract, mocks, version bumps, E2E pin). Load when touching internal/substrate, bumping Substrate, or triaging make test-e2e. metadata: version: "1.0.0"
Agent Substrate in AX
AX schedules every Task as a sandboxed actor on Agent Substrate. AX owns the declarative layer (Task/Workspace/Model, reconciler, CLI); Substrate owns isolation, checkpoint/restore, worker placement and in-cluster routing. Facts below were verified against main on 2026-11-06 — re-check go.mod and hack/setup-e2e-kind.sh if they look stale.
1. Module map
| Module | Pinned | Provides | Imported by | |
|---|---|---|---|---|
go.ate.dev/substrate | v1.0.0 | Control API protos api/ateapi/v1 (atespaces, actor templates, actors, tags, snapshot fidelity, suspend/resume, egress rules) | internal/substrate, internal/substrate/substratetest, internal/server, internal/controller, internal/apivalidation, test/e2e, pkg/apis/v1alpha1 (its .proto files import ateapi.proto) | |
github.com/agent-substrate/env | v0.0.11-0.20260912052224-4468a200b170 (pseudo-version) | Guest services served inside the sandbox (process exec + file system gRPC) and proto/ateenv/v1alpha | internal/guest, internal/metadata |
There is no replace directive for either module. Local hacking against a Substrate checkout goes through SUBSTRATE_DIR in E2E (section 5).
2. The Control API wrapper: internal/substrate.Client
All Control API traffic goes through this package. It encapsulates:
- Endpoint defaults —
api.ate-system.svc.cluster.local:443; authority
and TLS ServerName default to api.ate-system.svc when the target contains that string.
- Auth — bearer token from
SUBSTRATE_TOKEN_FILE(in-cluster: projected SA
token with audience api.ate-system.svc, see deploy/ax-server.yaml), CA from SUBSTRATE_CA_FILE.
- Retry —
IsTransientError+callWithRetry(retryOnCodes) forUnavailable-class errors. - Helpers the reconcilers rely on —
EnsureAtespace,EnsureActorTemplate,
EnsureActorTemplateWithImage, BuildActorTemplate, EnsureActor, EnsureActorFromTag, GetActorEgressPolicy, EnsureActorEgressPolicy, CreateTag, GetTag, ListTags, DeleteTag, SuspendActor, ResumeActor, DeleteActor, WorkerIP, ResourceLimits.
Where a new Control API call lives today
internal/substrate/client.gois the only non-test file that calls
ateapipb.NewControlClient; every existing RPC is a method on *Client, wrapped in callWithRetry (codes.Unavailable plus optional codes such as codes.Aborted).
- Each method has a table-driven case in
client_test.go(mockControlClient). - Callers in
internal/controllerexercise it throughmockControlServer;
internal/server and cmd/ax tests use internal/substrate/substratetest (section 4).
3. Address contract
| Value | Occurrences | |
|---|---|---|
api.ate-system.svc.cluster.local:443 + authority api.ate-system.svc | cmd/ax-server/main.go (--substrate-endpoint / --substrate-authority flag defaults), internal/substrate/client.go (3×), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, test/e2e/harness_test.go | |
atenet-router.ate-system.svc[.cluster.local] | internal/controller/reconciler.go (defaultRouterServerName), internal/tunnel/router_tls.go (routerServerName), cmd/ax/main.go (ssh port-forward path), internal/guest/client.go (doc comment), deploy/ax-server.yaml, test/e2e/manifests/ax-server.yaml, docs/networking.md | |
namespace ate-system | all of the above + test/e2e/harness_test.go (ateSystemNamespace), test/e2e/manifests/credential-policy.yaml | |
pod labels ate.dev/worker-pool (worker pods) and app: atenet-egress (egress gateway) | the ax-server NetworkPolicy in deploy/ax-server.yaml and test/e2e/manifests/ax-server.yaml, which keeps both away from ax-server | |
opentelemetry-collector.otel-system.svc:4317 (the collector Substrate's kind overlay deploys, manifests/ate-install/kind/ate-otel-config.yaml in Substrate) | test/e2e/manifests/ax-server.yaml (ax-otel-config ConfigMap). An unreachable address does not fail E2E; ax-server only logs OTLP export errors |
Why it matters: tasks never get a Service/Ingress. Every request to a task is proxied by the atenet router, which picks the target from the ate-target-actor header and resumes a suspended actor on demand (docs/networking.md). ax ssh port-forwards to svc/atenet-router on 443 (mTLS) and falls back to 80.
4. Testing against Substrate
Unit tests: in-process mock, no cluster
type mockControlServer struct {ateapipb.UnimplementedControlServer// record calls / inject failures here}grpcServer := grpc.NewServer()ateapipb.RegisterControlServer(grpcServer, mockSrv)// serve on a listener, point substrate.NewClient at it
Canonical examples: internal/controller/reconciler_test.go (reconciler state transitions; the largest test file in the repo). internal/server/server_test.go and cmd/ax/main_test.go run a real TaskReconciler against internal/substrate/substratetest, an in-memory fake of the same API (atespaces, actor templates, actors, and tags). internal/substrate/client_test.go uses a client-side mockControlClient instead. No mocking library is in go.mod.
Substrate behaviour the code already depends on
- Actor template images must be digest-pinned (
@sha256:…); a tag-only
spec.profile.image fails at ActorTemplate creation with reason TemplateCreationFailed (docs/manifests.md). make images writes bin/ax-server.digest / bin/ax-task-runner.digest for this reason.
- After
ResumeActorthe actor may be on a different worker, so AX never
stores a worker IP: the reconciler uses the IP ResumeActor returns only for its readiness probe (internal/controller/reconciler.go), and ax ssh connects through atenet-router.
- Egress: an actor without an egress policy has no outbound access. For a
task that binds a Gateway, bindGateway (internal/controller/reconciler.go) asks Substrate whether the actor has a policy. If it has none, bindGateway copies the Gateway's rules to the actor as its one policy (named default). An actor that has a policy keeps it, so the rules are copied once and never updated. bindGateway runs after the actor is suspended (CreateTask, SuspendTask) and before it is resumed (ResumeTask). Substrate validates the rules when they are copied. bindGateway fails with reason GatewayBindingFailed when Substrate refuses them, or when the actor has no policy and the Gateway cannot be loaded. SuspendTask and ResumeTask do not fail on the Gateway themselves (fetchGateway in internal/server/server.go), so a suspend always reaches SuspendActor. A task without a Gateway makes no egress policy call at all.
status.actorUidis read live bysetActorStatusin
internal/server/server.go and cleared by storedStatus before writes. It changes when Substrate recreates the actor and is empty when the actor is missing or cannot be read. ax describe task shows it as Actor UID.
- Guest services (process + fs) are mounted by
internal/metadata/server.go
only when task.Spec.Debug is true; ax ssh (cmd/ax/main.go) refuses to connect otherwise.
- Trust bundle:
BuildActorTemplatemountsegress-mitm.ate.devand
system-roots.ate.dev at v1alpha1.EgressTrustBundlePath for every task, and the runner points the CA variables at it (runner/env.go). A Substrate install that lacks either bundle blocks actor startup.
- Credential injection replaces a header only on requests that carry it. The
runner supplies a placeholder only for git: GIT_ASKPASS is /ax/git-askpass, a link to the running program, and package runner answers from init when a program that imports it is started under that name (runner/env.go). It sets no API key variable, because tools choose a provider by which key is set: a task sets its own placeholders in spec.env. A Gateway's spec is never passed into the sandbox.
- Readiness: Substrate and the reconciler probe the runner on port 80
(/healthz, /readyz?check=workspace, served by internal/metadata); /readyz is 503 until all workspaces are materialized (docs/sandbox.md).
- Snapshot storage: Actor templates created by AX leave
SnapshotConfig.StorageLocation
empty by default so Substrate uses its cluster-wide --snapshot-location. Specifying a custom snapshot bucket (snapshotsBucket in BuildActorTemplate) can only be used when the Substrate cluster has the PerTemplateSnapshotLocation preview feature gate enabled; otherwise Substrate rejects template creation with actor_template.snapshot_config.storage_location: Forbidden. Since v1.0.0 each path segment may contain only [A-Za-z0-9_-] and no segment may be atespaces; AX passes the value through unvalidated, so Substrate is the one that rejects it.
5. E2E and the Substrate version pin
test/e2e is compiled by go test ./... but self-skips unless AX_E2E=true or -args --e2e. The real runs go through make:
| Target | Substrate version | When | |
|---|---|---|---|
make test-e2e | exactly go list -m -f '{{.Version}}' go.ate.dev/substrate | PR CI (skipped when only paths-ignore files change, e.g. docs) and the nightly schedule (06:00 UTC) in e2e.yml | |
make test-e2e SUBSTRATE_REF=<release tag> | override | local, or workflow_dispatch input substrate_ref in e2e.yml | |
make e2e-setup / e2e-run / e2e-cleanup | same as above | iterate without re-provisioning |
E2E never builds Substrate. hack/setup-e2e-kind.sh installs the release's published images from ghcr.io/agent-substrate/substrate/<image>:<tag> (SUBSTRATE_IMAGE_REPO to use a mirror) via ATE_IMAGE_REPO/ATE_IMAGE_TAG, and pins the WorkerPool's worker image by digest. It exits early unless:
SUBSTRATE_REFis a release tag,vX.Y.ZorvX.Y.Z-rc.N. Branches,
commits, and Go pseudo-versions are rejected, and there is no run against Substrate main.
- The Substrate checkout is clean and at that tag. It still supplies the
manifests and ate-setup, so they must match the images.
- That release's
ate-setupsupports--image-repo, and every image the
install uses exists under the tag.
The install enables the k8s.io credential provider. For TestE2E_GatewayEgress the script then:
- Applies
test/e2e/manifests/credential-policy.yaml, which lets tasks in
atespace default use Secrets labeled ax.io/credential: "true" in the AX namespace, and restarts the provider, which reads its policy only at startup.
- Runs Substrate's
hack/setup-e2e-egress-tls-kind.sh, which makes the egress
gateway trust Substrate's service-DNS CA.
- Deploys the echo server (
test/e2e/manifests/echo.yaml), whose certificate
comes from that CA.
Useful env (full list in test/e2e/README.md): SUBSTRATE_DIR (local checkout; a detached worktree is created under .tmp/ if HEAD ≠ ref), SUBSTRATE_REPO, E2E_ARGS='-test.run TestE2E_ModelCRUD'.
6. Bumping Substrate
What the repo mechanically requires when go.ate.dev/substrate or github.com/agent-substrate/env changes in go.mod:
go get go.ate.dev/substrate@vX.Y.Z # and/or github.com/agent-substrate/env@<sha>go mod tidy # CI diffs go.mod/go.summake generate # buf reads ateapi.proto from the new versiongo build ./... && go test ./... # unit tests compile against the new ateapipbmake test-e2e # hack/setup-e2e-kind.sh installs the go.mod version
go.ate.dev/substrate must stay on a release tag: E2E installs that release's published images, and a pseudo-version fails E2E setup.
Gateway.spec.egress is Substrate's EgressRule, so a bump can change what users may write in a Gateway. Diff the egress messages in api/ateapi/v1/ateapi.proto between the two versions. If the new version adds a rule kind besides http, https, and tlsPassthrough, ValidateGateway (internal/apivalidation/gateway.go) rejects rules of that kind until its checks cover it. docs/manifests.md links to Substrate's egress credential injection guide at the pinned version.