version: "1.0.0" name: foundry-hosted-agent-validation description: > Step-by-step process for validating a Python Foundry hosted agent sample (under python/samples/04-hosting/foundry-hosted-agents/) end to end — running it locally (native runtime and azd ai agent run) and after deploying it to an Azure AI Foundry project with azd. Use this when asked to validate a hosted agent sample. license: MIT compatibility: Works with any model that supports tool use. metadata: author: agent-framework-samples version: "1.0"

Validating a Foundry Hosted Agent Sample

A hosted agent sample is "validated" when it passes three independent checks, plus cleanup:

1. Local, native runtime — run the sample's own entry point

(python main.py) and invoke it over HTTP.

1. Local, via `azd ai agent run` — the azd local dev loop.
2. Deployed — azd deploy to Foundry, then invoke the hosted agent.

Each check must succeed for single-turn and multi-turn (session / previous_response_id) conversation. Always end with cleanup (delete the deployed agent, remove the temp azd project, restore the sample dir).

Read the sample's own README.md and the parent

.../foundry-hosted-agents/README.md first — they define the run/deploy

commands and any sample-specific payload. This skill captures the process and

the non-obvious gotchas the READMEs don't.

Inputs you need before starting

Gather these:

• Foundry project endpoint, e.g.

https://<account>.services.ai.azure.com/api/projects/<project>.

• Foundry project resource id (for non-interactive azd ai agent init):

/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<account>/projects/<project>. Find it with az cognitiveservices account list + the project name.

• A real, deployed model name in that project (e.g. gpt-4.1-mini). This is

often different from the model id in agent.manifest.yaml — the actual deployment name wins.

• An existing ACR to reuse for deployment (login server, e.g.

myacr.azurecr.io). Reusing one avoids azd provision creating resources.

• Whether a like-named agent already exists in the project (remove it first

for a clean validation — see below).

Tooling / auth

• az (logged in: az login) and azd (logged in: azd auth login).
• azd agents extension: azd extension list should show

azure.ai.agents; install with azd extension install azure.ai.agents.

• uv for the native-Python local run. `python` need not be on PATH — uv

and azd ai agent run provision their own interpreter.

• Docker is not required when you reuse an ACR (remoteBuild: true builds

in ACR Tasks).

Phase 0 — Understand the sample

A responses/invocations sample folder typically contains: main.py (entry point + ResponsesHostServer/InvocationsHostServer), agent.manifest.yaml (used by azd ai agent init), agent.yaml (the deployed agent definition), requirements.txt, Dockerfile, .env.example.

The sample is the whole directory whose entry point is `main.py` — not every `.py` file in it. Other Python files in (or alongside) a sample folder are helper/companion scripts, not standalone samples. Do not treat a helper script as an individual sample — validate the sample via its main.py host, and run a helper only when the sample's README.md calls for it as a setup.

Note the protocol (responses or invocations) from agent.yaml / manifest — it changes the invoke command (--protocol invocations) and the HTTP path (/responses vs the invocations route).

Phase 1 — Local validation, native runtime (Python)

Run from the sample directory.

uv venv .venv --python 3.12          # 3.12 matches the sample Dockerfile
uv pip install --python .venv/... -r requirements.txt

Create .env from .env.example with the real values:

FOUNDRY_PROJECT_ENDPOINT="https://<account>.services.ai.azure.com/api/projects/<project>"
AZURE_AI_MODEL_DEPLOYMENT_NAME="<real-deployed-model>"

Start the server (python main.py) — it listens on http://localhost:8088. main.py uses DefaultAzureCredential, so az login must be current.

Invoke (single turn), capture the returned response_id, then reuse it for a follow-up turn to confirm memory:

curl -X POST http://localhost:8088/responses -H "Content-Type: application/json" \
  -d '{"input": "My name is Tao. Remember it."}'
# take response_id from the JSON, then:
curl -X POST http://localhost:8088/responses -H "Content-Type: application/json" \
  -d '{"input": "What is my name?", "previous_response_id": "<response_id>"}'

PowerShell: use Invoke-WebRequest -Uri http://localhost:8088/responses -Method POST -ContentType application/json -Body '...'.

Pass: HTTP 200, non-empty output[].content[].text, and the second turn recalls the name. Stop the server afterward.

Recording the playbook (cached replay)

The sample-validation harness caches a playbook so future runs replay your result without an agent. For a hosted-agent sample the playbook is an agent-authored Python script that reproduces only this Phase 1 native-local smoke test — never the azd/deploy phases (those are credential- and deployment-bound, non-deterministic, and must not enter the replay cache).

Emit a self-contained script that the harness runs with the active interpreter from the python/ directory (exit code 0 = success). It may import only the sample's own installed deps, the Python stdlib, and httpx. It must:

1. Start the sample server in the background with

subprocess.Popen([sys.executable, "<sample>/main.py"]) (path relative to python/; the shared python/.env and the process env supply credentials).

1. Poll http://localhost:8088 until the port accepts a connection (bounded

readiness loop with a timeout), failing if it never comes up.

1. POST turn 1 to the protocol's route (/responses for responses samples;

the invocations route for invocations samples), capture the response_id, then POST turn 2 with previous_response_id to confirm recall.

1. Assert HTTP 200, non-empty output[].content[].text, and that turn 2

recalls the fact from turn 1.

1. Always terminate the server in a finally block so port 8088 is freed

(the harness force-kills the process group as a backstop, but the script must still clean up). On any failure, print the captured server output before exiting non-zero.

Keep the deep three-phase validation (below) for a full manual/azd pass; it is out of scope for the cached playbook.

Phase 2 — Local validation via azd ai agent run

Init the azd project (once)

Run in an empty temp directory outside the repo (short path avoids Windows path-length issues, e.g. C:\afval\<sample>). Point -m at the local manifest so it validates the working-tree sample:

azd ai agent init -m <path>/agent.manifest.yaml \
  --project-id "<project-resource-id>" \
  --model-deployment "<real-deployed-model>" \
  --agent-name "<agent-name-from-manifest>" \
  --no-prompt --force

init downloads the template into a subfolder named after the agent, so the azd project root is <tempdir>/<agent-name>/. cd there for all later azd commands.

Before init, remove any `.venv` you created in the sample dir — init

copies the entire manifest directory into src/. (.venv is excluded from

deploy packaging by .agentignore/.dockerignore, so it is harmless but

bloats/slows the copy.)

Fix the model deployment name (critical — see Gotcha 1)

azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME "<real-deployed-model>"

Run and invoke locally

azd ai agent run --no-inspector        # auto-creates a uv venv + installs deps; listens on :8088
azd ai agent invoke --local --new-session "My name is Tao. Remember it."
azd ai agent invoke --local "What is my name?"   # same session is reused automatically

Pass: both invokes return text; the second recalls the name (same Session: id). Stop the run process afterward.

Removing a pre-existing agent (do this before deploying)

init prints a warning if the agent name already exists in the project. To delete it, note that `azd ai agent delete`/`show` resolve the deployed agent name from an azd env var, not from the positional argument. The var is AGENT_{SERVICEKEY}_NAME, where SERVICEKEY = the azure.yaml service name uppercased with -/spaces → _.

Example for service agent-framework-agent-basic-responses:

azd env set AGENT_AGENT_FRAMEWORK_AGENT_BASIC_RESPONSES_NAME agent-framework-agent-basic-responses
azd ai agent delete <service-name> --force --no-prompt --output json
# -> {"object":"agent.deleted","name":"...","deleted":true}

(After a successful azd deploy, this var is set automatically, so later show/delete/invoke work without setting it.)

Phase 3 — Deploy and validate

Reuse an existing ACR (avoid provisioning)

For an existing project + model, do not run `azd provision`/`azd up` — the generated azure.yaml has a deployments block for the manifest's model (often an auto-selected GlobalProvisionedManaged PTU SKU) that provision would try to create (costly / quota failures). Instead reuse an ACR:

azd env set AZURE_CONTAINER_REGISTRY_ENDPOINT <acr-login-server>   # e.g. myacr.azurecr.io
azd deploy

azd deploy fails with _"could not determine container registry endpoint"_ if this is unset and no ACR is provisioned.

Verify the deployed model env var, then invoke

azd ai agent show <agent-name> --output json   # check definition.environment_variables.AZURE_AI_MODEL_DEPLOYMENT_NAME
azd ai agent invoke <agent-name> --new-session "My name is Tao. Remember it."
azd ai agent invoke <agent-name> "What is my name?"

Use --output raw on invoke to see raw SSE events and any failure, e.g.:

event: response.failed
... "code": "DeploymentNotFound" ... 404 ...

DeploymentNotFound means the deployed AZURE_AI_MODEL_DEPLOYMENT_NAME points at a model that isn't deployed → fix per Gotcha 1 and redeploy (creates a new version).

Pass: agent reaches status: active, invoke returns text (not empty, no response.failed), and multi-turn recalls the name.

Cleanup (always)

• Delete the deployed agent: azd ai agent delete <agent-name> --force --no-prompt.
• Delete the temp azd project directory.
• Remove .env/.venv you created in the sample dir; confirm the sample dir is

pristine (git status --porcelain <dir> is empty — .env/.venv are gitignored).

• Stop any leftover local server still holding port 8088:

Get-NetTCPConnection -LocalPort 8088 -State Listen → Stop-Process -Id <pid> (Linux/macOS: lsof -ti:8088 | xargs kill). Stopping the shell may leave the child interpreter running.

Gotchas (the parts that waste the most time)

1. The deployed model name comes from `agent.yaml`, not the azd env.

azd ai agent init ignores --model-deployment in --no-prompt mode and writes the manifest's model id (e.g. gpt-4.1-mini) as a literal into both the azd env AZURE_AI_MODEL_DEPLOYMENT_NAME and the generated src/<agent>/agent.yaml env var. Local runs read the azd env (so azd env set fixes them), but deployment injects `agent.yaml`'s value. Fix by setting the generated agent.yaml env var to the template value: ${AZURE_AI_MODEL_DEPLOYMENT_NAME} (what the repo sample already uses; init flattens it) and azd env set AZURE_AI_MODEL_DEPLOYMENT_NAME <real>, then redeploy. A literal value: <real-model> also works.

1. `azd ai agent delete/show` need the `AGENT_{SERVICEKEY}_NAME` env var — a

bare positional agent name is treated as the _service_ name and the deployed agent name is looked up from that env var (see "Removing a pre-existing agent").

1. **azd provision/azd up will try to create the manifest's model

deployment** (from azure.yaml's deployments block). Prefer azd deploy with a reused ACR when the project + model already exist.

1. `python` on PATH is not required. uv venv and azd ai agent run

provision their own interpreter and install requirements.txt.

1. `init` copies the whole manifest directory into `src/`. Remove a local

.venv from the sample dir first to keep the copy clean/fast.

1. Port 8088 can stay bound after stopping the shell — kill the interpreter

by PID (see Cleanup).

Success checklist

• [ ] Native local run: 200 + non-empty text + multi-turn recall.
• [ ] azd ai agent run local: text returned + session reused across invokes.
• [ ] Pre-existing agent removed (if any).
• [ ] azd deploy succeeds; agent status: active.
• [ ] azd ai agent show confirms AZURE_AI_MODEL_DEPLOYMENT_NAME = the real

deployed model.

• [ ] Deployed invoke: text returned (no response.failed) + multi-turn recall.
• [ ] Cleanup done: agent deleted, temp project removed, sample dir pristine,

port 8088 free.