hermesonehq/hermes-registry/agent-platform-alert-configuration

Agent Platform Alert Configuration

This skill provides dynamic threshold alerting configurations for Google Cloud / Vertex AI Reasoning Engines (Agent Platform container deployments) using extended 1-week lookback retention baselines. Standard static thresholds (e.g., "latency > 2s") cause excessive alert noise for AI agents. Dynamic PromQL baselines solve this.

Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or writing configurations on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-only (`check_telemetry.py`)

• Rule: No confirmation needed. You may execute these scripts

immediately to inspect the telemetry status of the Reasoning Engine.

1. **Tier B: Billing & Resource Creation (create_online_monitor.py /

provisioning)**

• Rule: Explicit User Confirmation Required. These actions incur

additional billing charges and create cloud resources. The agent MUST ask the user directly for approval before proceeding.

CRITICAL RULES

• Always configure both Reliability and Quality alerting policies for the

target agent (6 policies in total):

• For Reliability Monitoring: You MUST configure exactly three

alerting policies:

1. Latency (anomaly monitoring)
2. Error Rate - Fast Burn SLO (1-Hour Window)
3. Error Rate - Slow Burn SLO (3-Day Window)

• For Quality Monitoring: You MUST configure exactly three alerting

policies:

1. Final Response Quality
2. Tool Use Quality
3. Hallucination

• Online Monitor Provisioning & Cost Warning: Quality alerting policies

rely on metrics exported by Online Monitors. You MUST ensure the Online Monitor is provisioned for the agent and telemetry is enabled:

• [ ] Ask for Approval: Both Online Monitors and Telemetry incur

separate billing charges. Before provisioning them, you MUST warn the user about these extra costs. If not pre-approved in the prompt, you MUST ask a direct question in your response requesting confirmation/approval to proceed (e.g., "Please confirm if you approve the extra billing costs for the Online Monitor and Telemetry to proceed.").

• [ ] Verify Telemetry First: Before generating any alerting policy

plan or provisioning Online Monitors, you MUST always verify the telemetry status of the Reasoning Engine first using the check_telemetry.py script as detailed in Verify Telemetry Status below.

• [ ] Follow the Guide: Follow the step-by-step instructions in the

Online Monitor & Telemetry Provisioning section below.

• Brand New Agents (No Traffic History): When setting up alerts for a

brand new agent, you MUST explicitly ask the user what traffic pattern they expect (Steady, Seasonal, or Bursty) in your response. If immediate setup is requested, ask the question but proceed using the default Steady/Consistent (Short-Window Z-Score) pattern. Follow no_historical_traffic_data.md.

• PromQL for Reliability (No MQL or Threshold Filters): For the 3

reliability metrics, you MUST use condition_prometheus_query_language with PromQL. Do NOT use MQL or standard condition_threshold.

• Standard Threshold Filters for Agent Quality: For the 3 agent quality

metrics, you MUST use standard condition_threshold filters matching the monitored resource type aiplatform.googleapis.com/OnlineEvaluator and metric type aiplatform.googleapis.com/online_evaluator/scores. Do NOT use PromQL.

• Install Terraform if Necessary: You should use terraform to deploy and

must install terraform if you can't find a valid install.

• Terraform Only: Write the generated observability configuration ONLY as

Terraform (.tf) files (e.g., alerts.tf, variables.tf).

• Dynamic Multi-Resource Alerting (No Single-Resource Pinning): You MUST

NOT hardcode specific agent IDs or resource name filters (e.g., {reasoning_engine_id="[AGENT_ID]"} or metric.labels.agent_resource_name="[AGENT_NAME]") in alerting conditions unless explicitly requested. Alerting policies must be written to cover all active agents in the project dynamically:

• For Reliability Metrics using PromQL: ALWAYS use grouping

aggregations (by (reasoning_engine_id)) instead of filtering to a single ID. This allows a single alert policy to dynamically track each reasoning engine instance separately.

• For Quality Metrics using Standard Threshold Filters: Omit the

agent_resource_name filter entirely. Configure the condition filter to only target the monitored resource type (aiplatform.googleapis.com/OnlineEvaluator) and metric type (aiplatform.googleapis.com/online_evaluator/scores) globally for the project.

• Check for Pre-existing Policies: Avoid creating duplicate alert policies

for a reasoning engine: scan the target directory or workspace to see if a policy already exists that targets the same metrics using aggregations grouped by reasoning_engine_id.

• Metric Scope Discovery & Project Inference: Centralize alert policies in

a Metric Scope (scoping project) to save costs. Identify if a scope is used and where policies should live by checking:

1. GCP CLI Check: Run `gcloud beta monitoring metrics-scopes list

projects/[PROJECT_ID]. If a parent scope locations/global/metricsScopes/[SCOPING_PROJECT_ID]` is returned, a Metric Scope is active; deploy policies there.

1. Infrastructure as Code Scan: Search Terraform configurations for

google_monitoring_monitored_project resources and extract the scoping project from the metrics_scope attribute.

1. Ambiguity Fallback: If unable to determine, ask the user: "Are you

using a multi-project Cloud Monitoring Metric Scope? If so, what is the scoping project ID?" Deploy policies to the deduced scoping project (setting the project attribute in HCL), or default to the local project.

• Directory Inference: Deploy configuration files to target Terraform or

SRE folders (e.g. monitoring/, ops/, sre/). Use tools to locate where alert policies or state pointers exist in the project, rather than blindly writing to the current working directory.

• Notification Channels: By default, never configure any notification

channels without user input. If the user explicitly provides a notification channel in their prompt, configure the alerts to use it. If no notification channel is provided, you MUST explicitly ask the user in your final response if they would like to configure notification channels. This is a mandatory question and you MUST NOT omit it from your response. IMPORTANT Do NOT make assumptions about notification channels. If you search the codebase for a notification channel you must ALWAYS confirm with the user before using it.

• Plain English Response: You MUST include a plain English explanation for

what the alerts do in your response. This must explain in plain English what the alert measures, how the algorithm works, and what a trigger indicates.

• Avoid Recursive Directory Operations: You MUST NOT run recursive listing

or search commands (such as ls -R, find ., or raw recursive grep) from the google3 workspace root, as this will hang your session. Always target specific subdirectories.

• Background Task Cleanup: You MUST check the status of all background

tasks that you spawn. Before completing your execution and returning your final response, you MUST terminate or kill any active or hanging background tasks (using the manage_task tool with action kill).

Algorithm Selection & Policy Mapping Process

Alerting policies for reasoning engine agents MUST map to the correct algorithms to ensure statistical stability and prevent alert noise or blind spots based on data classes:

• Latency: Follows workload traffic pattern (Steady -> Z-Score; Seasonal

-> Seasonal Decomposition; Bursty -> Moving Averages).

• Error Rate: ALWAYS use Multi-Window Multi-Burn Rate SLOs (or

ratio-based static thresholds). Error rate is naturally sparse (normally 0). When standard deviation is 0, Z-score computation is mathematically unstable (division-by-zero or NaN), causing false alert storms.

To resolve the workload traffic pattern (Seasonal, Steady, or Bursty), follow the instructions corresponding to the availability of historical metrics data:

• Case 1: No historical metrics data available (e.g., brand new agent):

You MUST read and follow: no_historical_traffic_data.md

• **Case 2: Historical metrics data available (e.g., active agent with

traffic)**: You MUST read and follow: has_historical_traffic_data.md

Telemetry Metrics and PromQL Examples

All raw telemetry metrics for the Agent Platform are cumulative counters. Because we monitor their rates or quantiles, we can optimize the PromQL queries by using longer range windows (e.g., [1w]) for historical averages instead of expensive avg_over_time subqueries.

Signal | Raw Metric | Type | Description :------------- | :------------------------------------------ | :------ | :---------- Latency | reasoning_engine_request_latencies_bucket | Counter | Histogram bucket of request latencies Error Rate | reasoning_engine_request_count | Counter | Cumulative count of requests

For the specific PromQL queries corresponding to each algorithm, you MUST read and follow: promql_queries.md

Agent Quality Metrics (Online Monitor)

All agent quality evaluation metrics are exported by Online Monitors to the monitored resource type aiplatform.googleapis.com/OnlineEvaluator under the metric type aiplatform.googleapis.com/online_evaluator/scores.

Metric Details & Aligners

Because the scores metric is of value type DISTRIBUTION, standard mean-based PromQL or arithmetic ALIGN_MEAN aligners are unsupported. You MUST use a percentile aligner (typically ALIGN_PERCENTILE_50 to evaluate the median score) within the aggregations block of your condition_threshold.

Signal | Metric Name (evaluation_metric_name) | Target Threshold | Recommended Aligner :------------------------------- | :------------------------------------- | :------------------ | :------------------ Final Response Quality | final_response_quality_v1 | < 0.8 (or custom) | ALIGN_PERCENTILE_50 Tool Use Quality | tool_use_quality_v1 | < 0.8 (or custom) | ALIGN_PERCENTILE_50 Hallucination (Groundedness) | hallucination_v1 | < 0.9 (or custom) | ALIGN_PERCENTILE_50

Metric Filter Example

When configuring a quality alert policy in Terraform, use the following filter expression structure: