arthur-ai/arthur-engine/arthur-onboard-eval-provider

Arthur Onboard — Step 8: Configure Eval Model Provider

Read State

cat .arthur-engine.env 2>/dev/null || echo "(no state file)"

Parse ARTHUR_ENGINE_URL and ARTHUR_API_KEY.

Check Existing Providers

curl -s \
  -H "Authorization: Bearer $ARTHUR_API_KEY" \
  "$ARTHUR_ENGINE_URL/api/v1/model_providers" | \
  python3 -c "
import sys, json
d = json.load(sys.stdin)
for p in d.get('providers', []):
    print(f'{p[\"provider\"]}: enabled={p[\"enabled\"]}')
"

If an enabled provider exists: Use it (prefer OpenAI > Anthropic > Gemini > Bedrock > Vertex AI). Write ARTHUR_EVAL_PROVIDER and ARTHUR_EVAL_MODEL to the state file, then exit this skill.

If No Provider Enabled

Ask user to choose:

• openai → model gpt-4o
• anthropic → model claude-3-5-haiku-20241022
• gemini → model gemini-1.5-flash
• bedrock → model anthropic.claude-3-haiku-20240307-v1:0
• vertex_ai → model gemini-1.5-flash
• Skip → configure in the Arthur Engine UI later (write ARTHUR_EVAL_PROVIDER=none to state, then exit)

If a provider is chosen, show the user this message and wait for their confirmation (do not ask them to type the key in chat; do not run getpass via the Bash tool — it has no TTY):

Please run this to securely enter your <Provider> API key (replace <Provider> with the actual provider name in the prompt):

! python3 -c "import getpass, os, stat; p=os.path.expanduser('~/.ae_tmp_key'); key=getpass.getpass('<Provider> API key (hidden): '); open(p,'w').write(key); os.chmod(p, 0o600); print('Key saved.')"

Let me know when done.

Then read the key from the temp file and configure:

PROVIDER_API_KEY=$(cat ~/.ae_tmp_key && rm -f ~/.ae_tmp_key)
curl -s -X PUT \
  -H "Authorization: Bearer $ARTHUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"api_key\": \"$PROVIDER_API_KEY\"}" \
  "$ARTHUR_ENGINE_URL/api/v1/model_providers/$PROVIDER"

Write ARTHUR_EVAL_PROVIDER and ARTHUR_EVAL_MODEL to .arthur-engine.env:

STATE_FILE=".arthur-engine.env"
grep -v '^ARTHUR_EVAL_PROVIDER=\|^ARTHUR_EVAL_MODEL=' \
  "$STATE_FILE" 2>/dev/null > /tmp/ae_env_tmp && mv /tmp/ae_env_tmp "$STATE_FILE" || true
echo "ARTHUR_EVAL_PROVIDER=$PROVIDER" >> "$STATE_FILE"
echo "ARTHUR_EVAL_MODEL=$MODEL" >> "$STATE_FILE"