alterlab-ieu/alterlab-academic-skills/alterlab-qualitative-analysis

Qualitative Analysis — Reliability When It Fits, Consensus When It Doesn't

Skill type: ANALYSIS MODULE. Codes qualitative data and reports agreement correctly. The central discipline is a branch: a reliability coefficient is right for codebook/content-analytic work, but for reflexive thematic analysis the criterion is consensus and reflexivity, not a statistic. The skill does not oversell the number.

Core Mission

KRIPPENDORFF'S ALPHA IS THE PRIMARY COEFFICIENT — BUT A COEFFICIENT IS THE WRONG TOOL FOR REFLEXIVE TA.
BRANCH BY DESIGN.

When to Use This Skill

• "Two RAs coded my interviews with a codebook — compute intercoder reliability."
• "Build/refine a codebook and report agreement."
• "I used an LLM to code open-ended responses — does it agree with my human coding?"
• "Which reliability statistics do I report for a content analysis?"

Does NOT Trigger

The branch (this is the core discipline)

Do not force α onto reflexive TA, and do not skip α on content-analytic coding.

Correct coefficient choice (verified)

• Krippendorff's α — primary. Handles ≥2 coders, any measurement level (nominal/ordinal/

interval/ratio), and missing data. Python krippendorff.alpha(reliability_data=<coders × units>, level_of_measurement="nominal") returns a point estimate only — pair it with a bootstrap CI (the bundled scripts/icr.py does nominal α + a bootstrap CI + Cohen's κ in pure stdlib; R irrCAC::krippen.alpha.raw or icr::krippalpha(..., bootstrap=TRUE) give CIs for all levels).

• Cohen's κ (2 coders) / Fleiss' κ (>2, fixed number): statsmodels.stats.inter_rater

cohens_kappa, fleiss_kappa (build the count table with aggregate_raters); sklearn cohen_kappa_score as a cross-check.

• NOT valid ICR measures: chi-square, Cronbach's alpha, and Pearson's r — they measure

covariation/internal consistency, not chance-corrected agreement. Refuse these.

Thresholds (Krippendorff): α ≥ .80 reliable; .667 ≤ α < .80 tentative conclusions only; α < .667 unreliable. Report α with a 95% CI, never the bare point estimate.

LLM-assisted coding (supported, never free)

Treat an LLM as another coder: human↔LLM double-coding, compute Krippendorff's α between each LLM and a human gold standard, and use the CI to judge whether the LLM is statistically indistinguishable from human coders. Disclose the model, the prompt, and the agreement — never present LLM coding as uncontrolled or cost-free.

Reporting checklist (put in every ICR report)

BRANCH:        content-analytic (coefficient) | reflexive TA (consensus+reflexivity)
ICR MEASURE:   Krippendorff alpha (level) | Cohen/Fleiss kappa — and why
CODERS:        number of coders; human / LLM
DOUBLE-CODED:  % of the data double-coded
ALPHA:         point estimate + 95% CI + threshold verdict (.80 / .667)
RESOLUTION:    how disagreements were resolved (adjudication / consensus)

References

• references/icr_and_reporting.md — coefficient math, verified APIs, LLM-benchmarking, the reporting standard.
• references/reflexive_vs_codebook.md — when a statistic applies vs. when it does not.
• scripts/icr.py — stdlib nominal Krippendorff's α + bootstrap CI + Cohen's κ (no numpy / no deps).

Part of the AlterLab Academic Skills suite.