vectorspacelab/arex-skill/agreement-statistics

Claude Code and agent skill by vectorspacelab/arex-skill

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Compute ID accuracy, pairwise ID/OOD agreement, and clipped probit statistics for agreement-on-the-line experiments.

Install

npx vskill@latest install vectorspacelab/arex-skill/agreement-statistics

SKILL.md

The same SKILL.md was published earlier as vectorspacelab/arex-skill/agreement-statistics.

Agreement Statistics

Use this skill after a prediction table has been validated. It preserves the paper's estimator boundary by computing OOD agreement from predictions only and never using OOD labels.

Inputs

  • Normalized prediction table from prediction_table_contracts.
  • Optional clipping epsilon for probit transforms.

Outputs

  • Per-model ID accuracies.
  • Pairwise ID and OOD agreement fractions.
  • Probit-transformed accuracies and agreements.

Workflow

  1. Compute ID accuracy from ID labels and ID predictions.
  2. Compute unordered pairwise agreement on ID predictions.
  3. Compute unordered pairwise agreement on OOD predictions.
  4. Clip probabilities before inverse-normal transformation.

Validation

Run python tests/test_agreement_statistics.py from this skill directory.

Limitations

This skill does not fit the agreement line or solve ALine-D.

Security

  • Security scanPassed, 90 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

7-day trend score
0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/vectorspacelab/arex-skill/agreement-statistics/badge)](https://verified-skill.com/skills/vectorspacelab/arex-skill/agreement-statistics)

Something wrong with this skill or its page? Tell us.