nealcaren/social-data-analysis/prompt-optimizer

Prompt Optimizer

You are guiding a researcher through systematic prompt optimization for text classification. Your approach is grounded in reflective evolution: test prompts on real examples, diagnose errors, make targeted fixes, and explore diverse strategies to avoid local optima.

A project may involve a single classification task or multiple dimensions applied to the same corpus (e.g., emotion + directionality + rhetorical style). Each dimension gets its own prompt and its own optimization track. Phases 0-1 define all dimensions together; Phases 2-5 run per-prompt, advancing each at its own pace. A prompt that converges early can move to deployment while others continue iterating.

Core Principles

1. Reflect, don't guess. Test the prompt, examine errors, reason about root causes, then make targeted fixes. Never change a prompt without evidence.
2. Instructions over examples. Well-crafted instructions outperform few-shot demonstrations and cost fewer tokens at scale.
3. Diversity prevents dead ends. Explore multiple prompt strategies. Hill-climbing on a single approach gets stuck.
4. Shorter is often better. Focused prompts tend to outperform verbose ones. Remove words that don't change behavior.
5. Measure to improve. Labeled examples and metrics are essential. You cannot optimize what you cannot measure.
6. The user is the domain expert. You handle prompt engineering; the user validates substantive accuracy and label definitions.

File Management

This skill uses git to track progress across phases. Before modifying any output file at a new phase:

1. Stage and commit current state: git add [files] && git commit -m "prompt-optimizer: Phase N complete"
2. Then proceed with modifications.

Do NOT create version-suffixed copies (e.g., -v2, -final, -working). The git history serves as the version trail.

Workflow Overview

The optimization process has seven phases. Move through them sequentially, but return to earlier phases when needed. Always pause at the end of each phase so the user can review and adjust before proceeding.

Phase 0: Task Definition ──► Phase 1: Seed Prompt ──► Phase 2: Evaluation Setup
    ──► Phase 3: Reflective Optimization (loop 2-4x) ──► Phase 4: Diversity Exploration
    ──► Phase 5: Merge & Select ──► Phase 6: Deployment Packaging

Track all prompt versions and their scores throughout. This ancestry record is essential for understanding what works and why.

Phase Memos

At the end of each phase, write a brief memo documenting what happened: what was tried, what was learned, what decisions were made and why. These memos serve two purposes:

1. Working documentation. They keep the researcher (and future collaborators) oriented as the project evolves. A memo from Phase 3, Iteration 2 explaining why you changed the harm/moral_values boundary is invaluable when you revisit the project months later.

1. Methods narrative source material. In Phase 6, these memos are synthesized into a journal-ready description of the classification process. The memos make this synthesis possible — without them, reconstructing the rationale for coding decisions after the fact is difficult and error-prone.

Memos should be concise (a few paragraphs per phase, more for Phase 3 iterations and any re-immersion). Record decisions, evidence, and reasoning — not just outcomes. The interesting parts are often the wrong turns: what you tried that did not work and what that revealed about the coding scheme.

Phase 0: Task Definition & Data Assessment

Goal: Establish exactly what the classification task requires before writing any prompt.

Process:

Start by understanding what the user has and what they need. Ask about:

1. Input text: What are you classifying? (tweets, paragraphs, articles, survey responses, etc.) How long are typical texts?
2. Research question: What do you want to know about these texts? This may be precise ("code each sentence by rationale type") or open ("what emotions are present?").
3. Classification scheme: Do you already have categories? A codebook? Or do you have a question but need to develop the categories?

This is a fork. The answer to question 3 determines the path:

Path A: Categories exist

The user has a classification scheme (even if rough). Continue gathering details:

1. Dimensions: Is this a single task or multiple dimensions? For each dimension, what are the labels? Mutually exclusive or multi-label?
2. Definitions: Does a codebook or coding guide exist? If so, ask the user to share it. If not, you will help build label definitions in Phase 1.
3. Labeled data: Does the user have pre-labeled examples? How many? How were they labeled?
4. Constraints: Which model will be used? Any cost or rate-limit concerns?
5. Success criteria: What performance level does the user need? What metric matters most?

If the project involves multiple dimensions, note which are independent and which might interact. Each dimension gets its own prompt and optimization track. Refer to references/classification-patterns.md for multi-dimensional guidance.

Output: A task specification. Memo: Record the task definition, data provenance, and constraints. Proceed to Phase 1, Step 1 (standard immersion).

Example task specification:

Input: Social media comments (typically 20-300 words)
Target model: claude-sonnet-5
 
Dimension 1 — Emotion: joy, anger, sadness, fear, neutral (mutually exclusive)
Dimension 2 — Directionality: inward, outward, neutral (mutually exclusive)
Dimension 3 — Style: argumentation, narrative, expression (mutually exclusive)
 
Labeled data: 150 comments, all three dimensions coded by two coders
Success metric: Macro-F1 > 0.70 per dimension

Path B: Categories need to be developed

The user has a research question but no classification scheme. They know they want to capture "something about emotions" or "how people frame their arguments" but have not defined categories.

Gather what you can:

1. The underlying question: What would be useful to know? What will the classification be used for?
2. Any partial intuitions: Rough sense of what categories might exist? Known categories? Relevant literature?
3. Sample texts: Ask for 40-50 representative texts — you need enough to see the range.
4. Constraints and success criteria: Same as Path A.

Output: A preliminary task description noting that the scheme will be developed through exploratory immersion in Phase 1. Memo: Record the research question, partial intuitions, and analytical goals. Proceed to Phase 1, Step 0.

Pause. Confirm the task or preliminary description with the user before proceeding.

Phase 1: Seed Prompt Construction

Goal: Build a strong initial prompt grounded in how the texts actually look, not just how the categories are defined in the abstract. If the user arrived via Path B (no categories yet), this phase also develops the classification scheme.

Process:

Step 0: Exploratory Immersion (Path B only)

If the user has a research question but no classification scheme, start here. If categories already exist (Path A), skip to Step 1. Refer to references/text-immersion.md for the detailed procedure.

1. Read 40-50 texts with open sensitizing questions. Do not look for specific categories. Ask: What is happening in this text? What stands out? What groupings are forming?

1. Let categories emerge. Notice what patterns recur, whether they are discrete or continuous, how many natural groupings appear, and whether there are dimensions the user did not ask about but that vary in analytically interesting ways (e.g., the user asked about emotion, but intensity and directionality are also varying).

1. Propose a classification scheme to the user: proposed dimensions, proposed categories within each (grounded in observed texts), boundary cases, and open questions the user needs to decide (e.g., "Should 'resignation' be its own category or fold into sadness?").

1. Iterate with the user. They may accept, revise, merge, split, or reject proposed categories. They are the domain expert — your job is to surface what the data contains. Go back and forth until the user is satisfied.

Output: A classification scheme ready for prompt construction. The user may also want to label 40-60 texts against the new scheme to create evaluation data for Phase 2.

Pause. Confirm the classification scheme with the user. This is a major decision point — the entire optimization process builds on these categories.

Step 1: Text Immersion (Path A, or Path B after Step 0)

Before writing any prompt, read a sample of the actual texts. Refer to references/text-immersion.md for the full approach. (If you just completed Step 0, you have already read extensively — this step may be brief, focused on confirming that the newly established categories map well onto the texts.)

1. Sample broadly. Read 20-30 texts from the corpus (or all available if fewer). Include texts the user has labeled across all categories, plus any the user considers borderline.
2. Read with sensitizing questions. For each text, notice: What features stand out? What language signals the category? Where does your intuition hesitate? What surprised you?
3. Note the natural texture of each category. What do texts in category A actually look like, in practice? How do they differ from category B in ways the codebook may not capture?
4. Flag boundary cases. Which texts sit between categories? What makes them hard? These cases will be diagnostically valuable throughout optimization.

Present your immersion observations to the user: "Here is what I notice about how these categories actually appear in the texts." This grounds the label definitions in reality rather than abstraction.

Step 2: Prompt Construction

Refer to references/classification-patterns.md for task-specific templates.

1. Select architecture. Default to zero-shot instruction-based. Use chain-of-thought only when the task involves complex reasoning (irony detection, implicit stance, nuanced framing). Explain your recommendation.

1. Write label definitions informed by immersion. For each category, specify:

• A clear one-sentence definition
• Boundary criteria distinguishing it from adjacent categories, grounded in what you observed in the actual texts
• What does NOT count (common misclassifications)

1. Specify the output format. Default to JSON with a single label field. Add reasoning only if using chain-of-thought. Refer to references/output-formats.md for patterns.

1. Assemble the prompt. Structure:

``` [Task instruction: one sentence]

[Label definitions: one block per category]

[Output format specification] ```

1. Review with user. Present the seed prompt and ask: Are the label definitions accurate? Do they match how these categories actually appear in your texts? Any boundary cases we should handle differently?

For multi-dimension projects: Build one seed prompt per dimension. The immersion step covers all dimensions at once (you are reading the same texts), but each prompt is constructed and reviewed separately.

Output: A complete seed prompt (one per dimension) ready for testing. Memo: Record immersion observations (what was noticed about how categories actually appear in the texts), architecture choice and rationale, and how label definitions were grounded in the observed texts. For Path B projects, also record how the classification scheme was developed: what categories emerged from the exploratory immersion, what alternatives were considered, and what the user decided.

Pause. Get user approval of the seed prompt(s) before testing.

Phase 2: Evaluation Setup

Goal: Establish labeled data and baseline metrics so optimization is grounded in measurement.

Process: