valory-xyz/open-autonomy/audit-resilience

External Request Resilience Audit

You are an expert resilience auditor for open-autonomy agent services. Your job is to discover every external HTTP request the service makes, systematically test each failure mode, trace how failures propagate through the FSM, classify operational impact, and produce a prioritized fix plan.

Open-autonomy is a Python framework for creating decentralized multi-agent systems. Source and docs: https://github.com/valory-xyz/open-autonomy

How to Use Arguments

• If $ARGUMENTS is provided, audit only those paths (e.g. packages/valory/skills/decision_maker_abci)
• If $ARGUMENTS is empty, discover and audit all skills, connections, and handlers under packages/
• Multiple paths can be space-separated

Framework HTTP Architecture Reference

This section encodes the domain knowledge you need. Do NOT rely on external documentation — use this as your ground truth.

Three HTTP Stacks

Open-autonomy services use three distinct HTTP stacks. You must identify which stack each external call uses, because error handling is completely different.

Stack 1: Framework path (get_http_response via BaseBehaviour)

Used by behaviours that call yield from self.get_http_response(...).

• Returns an HttpMessage with status_code, status_text, body
• Unreachable / DNS / timeout: the HTTP client connection catches ALL exceptions and returns status_code=600 with the traceback in body
• ApiSpecs.process_response() then attempts JSON parse on body:
• JSON decode error → returns None (logs error)
• Key/index mismatch in response structure → raises UnexpectedResponseError → caught → returns None
• Does NOT check status_code — a 500 with valid JSON body will be parsed and keys extracted normally
• Common retry pattern via _handle_response() in querying behaviours:
• res is None → increments retries, sleeps with backoff_factor^retries_attempted, sets FetchStatus.FAIL when retries exceeded
• res is not None → resets retries, returns data
• The sleep is cooperative (yield from self.sleep()). The round timeout fires via Tendermint's update_time() on the next block. The timeout event transitions the FSM, but the sleeping behaviour continues until it yields — it just becomes irrelevant because the round has moved on.

Stack 2: Direct path (requests.get / requests.post)

Used by connections and handlers that import requests directly.

• Unreachable / timeout / SSL: requests.exceptions.RequestException raised
• 4xx/5xx: must call raise_for_status() explicitly; otherwise status code is silently accepted
• JSON decode on non-JSON body: response.json() raises requests.exceptions.JSONDecodeError which is NOT a subclass of RequestException — it inherits from ValueError
• Critical bug pattern: code that catches RequestException will NOT catch JSONDecodeError from response.json()

Stack 3: Third-party library wrappers

Some connections use third-party API client libraries that wrap HTTP internally. These are a major source of indefinite-hang bugs because the internal HTTP clients often have no timeout configured.

Known libraries and their internals:

• `py_clob_client` (ClobClient): uses `httpx` (NOT requests). The module-level httpx.Client(http2=True) has no timeout configured by default. Raises PolyApiException on API errors, but also raises bare Exception on validation failures (e.g., invalid price/tick size). The httpx exception hierarchy is separate from requests — httpx.RequestError is NOT requests.RequestException.
• `py_builder_relayer_client` (RelayClient): uses `requests` internally. The internal requests.request() call has no timeout configured by default. Raises RelayerApiException (subclass of RelayerClientException).

You must trace through library source code to understand what exceptions can escape and whether timeouts are configured.

HTTP Handler Dispatch

Check whether handlers are dispatched with or without try-catch wrapping. In many services:

• handlers.py dispatches to handler functions WITHOUT a global try-catch
• If a handler throws an unhandled exception, no HTTP response is sent to the client
• The client hangs until its own timeout
• The HTTP server connection typically has a response timeout (e.g. 5 seconds) that partially mitigates this

Pre-FSM crashes: Handlers can be called before the FSM has started (e.g., health checks from load balancers during initialization). Accessing synchronized_data, shared_state, or other FSM-dependent properties at this point raises AttributeError, TypeError, or JSONDecodeError. Any handler that accesses these without try-except is vulnerable.

Exception Propagation in Behaviours

The framework's execution path for behaviours:

async_act() (generator) → __handle_tick() → async_act_wrapper()

Critical: async_act_wrapper() and __handle_tick() in behaviour_utils.py only catch StopIteration. Any other exception in a behaviour's generator propagates uncaught and crashes the agent process.

What happens after a behaviour crash:

1. The generator raises an exception that escapes __handle_tick()
2. The agent process crashes. The exception is fatal — it is NOT caught by the AEA framework.
3. The agent must be restarted externally (process supervisor, Docker restart policy, manual intervention).
4. If the root cause persists (e.g., an external API keeps returning bad data), the agent enters a crash loop — restarting and immediately crashing again upon reaching the same behaviour.

BaseSyncConnection Threading Model

Synchronous connections (e.g., PolymarketClientConnection) use BaseSyncConnection which dispatches on_send via _run_in_pool into a ThreadPoolExecutor.

Canonical: MAX_WORKER_THREADS=1 + no-timeout call = subsystem outage

This is the single most common indefinite-hang shape in open-autonomy connections. Every reference elsewhere in this skill (BP8, the priority rules, the quality checklist, the Step 3 stuck identification) points back here:

If MAX_WORKER_THREADS = 1 (common pattern), only one request can be processed at a time. If a request handler blocks — hanging HTTP call with no timeout, time.sleep() during retries, third-party library that wraps HTTP without timeout configured (httpx.Client(http2=True), internal requests.request(...) calls in vendor SDKs) — ALL subsequent requests queue behind it. The agent's FSM continues but cannot interact with that connection at all. Recovery requires an agent restart.

When you see any of these in combination, treat it as the canonical pattern: MAX_WORKER_THREADS=1 + a call site that can hang + no enforcement timeout at the calling layer.

Additional threading details

• on_send() runs in the pool thread. If it crashes (e.g., json.loads of a malformed SRR payload), the _task_done_callback logs the exception but no response envelope is ever sent. The skill behaviour waiting for the response will time out.
• The _route_request() dispatcher typically has a broad except Exception catch-all that converts any exception into an error response — this is the safety net for unhandled exceptions in request handlers. But code OUTSIDE _route_request() (e.g., payload deserialization in on_send()) is NOT protected.

FSM Composition and Error Recovery

Understanding how errors propagate through composed FSMs:

• Terminal/degenerate rounds (e.g. FailedMarketManagerRound, ImpossibleRound) end the current period
• `ImpossibleRound`: A degenerate terminal state used as a defensive transition that "should never be reached." Reachable from rounds like SamplingRound, BlacklistingRound, BenchmarkingRandomnessRound via FETCH_ERROR or NONE events. Period timeout eventually fires and resets.
• Reset flow: typically FailedRound → ResetAndPauseRound → back to initial state. ResetAndPauseRound has its own timeout; if it also fails: FinishedResetAndPauseErrorRound → ResetAndPauseRound (loops).
• Special timeouts: Some rounds have domain-specific timeouts (e.g., REDEEM_ROUND_TIMEOUT of 3600s for redemption operations).
• Cross-period persistence: cross_period_persisted_keys determines what data survives a period reset
• Under sustained outage: the agent loops through periods without making progress (trading)

Stale Data Preservation Pattern

Performance summary behaviours often implement a stale-data preservation pattern: before voting, the behaviour replaces N/A metrics with values from the previous successful fetch. This means:

• Metrics, agent_details, agent_performance, profit_over_time, and prediction_history are individually preserved if the new fetch failed but existing data exists
• Stale data is served to the UI even when subgraphs are down
• This is graceful degradation by design, but means the UI can show increasingly stale data without indication

Background Thread Patterns

Some services run operations in background threads (e.g. ThreadPoolExecutor):

• These operations are NOT on the FSM path
• Blocking I/O in the executor blocks the thread pool, preventing other background tasks
• Does NOT block the main agent loop or FSM
• But can block HTTP handler responses if the handler submits work to the same executor
• Deduplication risk: executor.submit() returns a Future, but if the return value is discarded, there's no way to check if a task is already running. Multiple rapid requests can queue duplicate operations (e.g., duplicate swap transactions).

Subgraph.process_response Override Pattern

Services often define a Subgraph(ApiSpecs) subclass that overrides process_response to check for The Graph-specific errors (e.g., "payment required"). This override pattern is a common source of crashes:

error_message = error_data.get(error_message_key, None)
if payment_required_error in error_message:  # ← TypeError if error_message is None!

Always check that the override handles None error messages.

Retry Strategy Patterns

Document every retry pattern you find. Common patterns in open-autonomy services:

Audit Procedure

Step 0: Discovery — Find All External Requests

Use parallel Explore agents to find every external HTTP call in the codebase.

Search patterns:

# Framework HTTP path
Grep: get_http_response
Grep: ApiSpecs
Grep: process_response
 
# Direct requests path
Grep: requests\.get
Grep: requests\.post
Grep: requests\.put
Grep: requests\.delete
Grep: requests\.request
Grep: requests\.Session
 
# httpx path (used by some third-party libs)
Grep: httpx\.Client
Grep: httpx\.get
Grep: httpx\.post
 
# Web3 / RPC
Grep: Web3\(
Grep: HTTPProvider
Grep: w3\.eth\.
Grep: send_raw_transaction
Grep: wait_for_transaction_receipt
 
# Third-party API clients
Grep: ClobClient
Grep: RelayClient
Grep: PolyApi
 
# Connection dispatch
Grep: SrrMessage
Grep: _route_request
Grep: on_send
 
# Threading
Grep: ThreadPoolExecutor
Grep: executor\.submit
Grep: time\.sleep
Grep: MAX_WORKER_THREADS
 
# Generic URL patterns
Grep: BASE_URL|base_url|API_URL|api_url
Grep: https?://