fergana-labs/stash/intuitions

Intuitions

An intuition is a classifier that matches one person's judgment call. Under the hood it is not a fine-tuned model:

• Jev, a judge model, answers each criterion (an observable yes/no or

multiple-choice question about the input) and the task itself, zero-shot. Every answer comes back as a probability.

• A small calibrated head learns from the person's labels how much each

answer matters. It starts from "trust the zero-shot answer", so 6–10 labels already help.

• An embedding probe learns what the labeled examples look like, beyond what

the criteria say.

• Accuracy is always measured on held-out examples the head never trained on.

Three kinds:

Connect

MCP (preferred): streamable HTTP at https://<your stash host>/mcp/intuitions, with header Authorization: Bearer <Stash API key>.

{"mcpServers": {"stash-intuitions": {"type": "http",
  "url": "https://joinstash.ai/mcp/intuitions",
  "headers": {"Authorization": "Bearer st_..."}}}}

REST: the same bearer token against /api/v1/intuitions and /api/v1/model-wizard. Classifying one input:

curl -s https://joinstash.ai/api/v1/intuitions/$ID/predict \
  -H "Authorization: Bearer $STASH_API_KEY" -H 'Content-Type: application/json' \
  -d '{"item": "text, or a JSON object", "ref": "your-id-for-feedback"}'

Build one well

Do these in order. Skip what the person already told you, and never ask more than two questions at a time.

1. One decision. What is decided, by whom, and what happens next? If the

person describes two decisions, build one now and offer the other next.

1. Real inputs first. Ask for one or two real examples before talking

about labels.

1. Labels with boundaries. Every label needs a definition and the nearest

thing it is not. Ask for the hardest borderline case and what the right call is; this is where most classifiers go wrong. If several things can be true of one input at once, use multi.

1. Stakes. Which mistake is worse? Which cases must it never get wrong?

Add those as labeled, pinned examples: no version can ship if it gets a pinned case wrong.

1. Brief. Record all of the above with save_brief. It is compiled into the

instructions the judge reads, so keep it accurate.

1. Examples. Get 8+ confirmed labels with every label present, mostly real

and hard. Real data first: ask where real cases live (exports, logs, tickets, past decisions). Only if the person can't provide them, or their data lacks a kind of case, use generate_examples aimed at that gap. Drafts come with a suggested label and reason, and a person must confirm them (label_example, or accept_suggestions only when they say so). Never invent labels yourself.

1. Criteria. 3–6 observable properties. Once both kinds of label exist,

propose_criteria with mode contrastive finds what separates similar items that were labeled differently. Save them with set_criteria.

1. Train, then report held-out accuracy with its sample size, next to

zero-shot Jev. With fewer than about 20 held-out examples, call it an early estimate.

1. Ship when the gate passes.

Use it

Keep it improving

• inbox lists recent decisions, least confident first. review_prediction

confirms or corrects one, with a reason. It becomes a golden example and can be pinned.

• give_feedback records downstream outcomes against a prediction_id or your

own ref.

• Reasons from corrections drive new criteria; train again after a batch of

corrections.

MCP tools

list_intuitions, create_intuition, get_intuition, update_intuition, save_brief, add_examples, generate_examples, list_examples, label_example, accept_suggestions, propose_criteria, set_criteria, train, ship, classify, inbox, review_prediction, give_feedback.