scdenney/open-science-skills/conjoint-design

Conjoint Design Expert

Worked example (attribute table → power calculation → PAP tier assignment): see references/example.md.

1. Is a Conjoint the Right Design?

• Identify the DGP Before Attributes: Articulate the data-generating process the theory implies: which component of the compound profile bears the causal effect of interest, and what respondents are assumed to do when bundling or separating information. This is the "Why → If-Then" step of hypothesis-building; the attribute table follows from it.
• When a Conjoint Is the Wrong Tool: Prefer another design when (a) the question concerns a single treatment dimension (a simple vignette experiment suffices), (b) the attributes cannot be independently randomized without producing incoherent profiles, or (c) the mechanism operates through holistic impression formation rather than attribute-by-attribute evaluation (Sniderman 2018).
• Between-Subjects to Conjoint Conversion: When a between-subjects vignette experiment suffers from confounds (outcomes not held constant across conditions), a conjoint (a) holds other features constant through independent randomization, (b) forces concrete content rather than vague descriptions, (c) gains power from repeated within-respondent measurements, and (d) enables estimation of interaction effects.

2. Attribute Architecture

• Independence: Randomize every attribute independently of every other so each component's causal effect is identified.
• Randomization of Order: Randomize attribute order at the respondent level, not the task level, to prevent primacy and recency effects without making respondents search for information in a new order on every task (Stantcheva 2023). A theoretically required logical flow may override this.
• D-Optimal Designs: Consider D-optimal or constrained randomization rather than pure randomization when the number of possible attribute combinations is large relative to the sample; these choose the administered conditions that maximize power (Auspurg & Hinz 2015; Stantcheva 2023).
• Attribute and Task Density: Bansak et al. (2018) find response quality does not degrade up to 30 tasks on MTurk; Bansak et al. (2021, "Beyond the Breaking Point") extend this to the attribute dimension; Stefanelli and Lukac (2020) cite stability up to 10 attributes. Still judge whether the complexity of the levels, not just their count, raises cognitive load.
• Nested/Constrained Randomization: Attributes need not all be fully crossed. Linking or nesting levels (e.g., origin countries nested within policy domain) is acceptable when (a) the nesting is theoretically justified, (b) the primary attributes of interest remain fully independently randomized, and (c) the analyst acknowledges that nested attributes cannot be cleanly separated from their parent (Auspurg & Hinz 2015).
• Attribute-Level Restrictions: Excluding implausible combinations is a judgment call, not a mandate. Eye-tracking evidence shows odd (incongruent or nonsensical) combinations have minimal, inconsistent effects on attention, search, and choice (Bansak & Jenke 2025), so decide on statistical, substantive, and theoretical grounds — e.g., whether the randomization distribution should reflect a real-world target distribution (De la Cuesta, Egami & Imai 2022) — rather than assumed cognitive burden. Document all restrictions in the PAP.
• Medium-Level Specificity: Describe levels "fully described but not overly described" (Sniderman 2018): concrete enough to be meaningful, not so specific that they introduce confounds. Neither "a policy that helps the economy" nor "a $2.3B infrastructure bill for Route 95 in Pennsylvania."

3. Statistical Power and Error Logic

• Effective N: N_eff = Respondents × Tasks × Profiles (the number of profile evaluations). Respondents and tasks are not interchangeable: within-respondent correlation means adding respondents improves precision more than adding tasks, so prioritize respondents (Stefanelli and Lukac 2020).
• Closed-Form Formula: SE(AMCE) ≈ $\sqrt{\text{Var}(Y) \times L / N_{\text{eff}}}$, where $L$ is the attribute's number of levels (Schuessler and Freitag 2020). Use it as a quick precision diagnostic.
• Interaction Power: In the balanced two-level-by-two-level case the interaction SE is about $\sqrt{2}$ times the main-effect SE, so interactions need roughly twice the sample of main AMCEs; the multiplier grows with the number of levels on each interacting attribute (Schuessler and Freitag 2020). Budget for this when interaction hypotheses are confirmatory.
• Empirical AMCE Benchmarks: Published AMCEs typically range from 0.02 to 0.10 (percentage-point changes in selection probability), median around 0.05; AMCEs above 0.15 are rare (Stefanelli and Lukac 2020). Use these to set the SESOI when no prior data exist.
• Cohen's d Does Not Apply: AMCEs are in percentage points of choice probability, not standard-deviation units; translating requires Var(Y), which depends on the task structure. Do not calibrate conjoint power with small/medium/large d benchmarks.
• Minimum Detectable Effect: Power the hardest contrast — the attribute with the most levels and the smallest expected AMCE — and report whether the MDE falls within the range of plausible AMCEs from prior literature.
• Levels-Power Tradeoff: Each added level reduces per-level N (4 → 5 levels cuts it by about 20%) with a corresponding precision loss (Schuessler and Freitag 2020). Add a level only when it is theoretically necessary.
• Low-N_eff Danger Zone (rule of thumb): Below ~3,000 effective profile evaluations, designs are at high risk of being underpowered for typical AMCEs (0.02–0.05). This threshold comes from plugging median benchmarks into the Schuessler and Freitag (2020) formula, not from a research finding; adjust for expected magnitude, levels, and design. Below it, report a sensitivity analysis of what effects can be detected.
• Type S and Type M Errors: At low power, significant estimates risk the wrong sign (Type S) and exaggerated magnitude (Type M), and both worsen steeply as power falls (Gelman and Carlin 2014). With a two-sided test at α = .05, a significant estimate overstates the true effect by about 1.4 times on average at 50% power, with a negligible sign risk; at 17% power the exaggeration is about 2.5 times and about 1 in 110 significant estimates has the wrong sign; at 8% power, about 4.8 times and 1 in 11. Base the design analysis on a plausible effect size from prior conjoints or theory, not on a pilot's noisy estimate.
• Multiple Testing: A conjoint estimates many AMCEs at once. Under the sharp null, a typical pipeline yields at least one significant AMCE in more than 90% of trials (Liu and Shiraito 2023) — the forking-paths and undisclosed-flexibility problem (Gelman and Loken 2014; Simmons, Nelson, and Simonsohn 2011). Pre-specify a correction across the full AMCE family: Bonferroni (most conservative), Benjamini-Hochberg (controls FDR, most lenient), or adaptive shrinkage (balanced; preferred in Liu and Shiraito's simulations). Report corrected and uncorrected results for confirmatory hypotheses.
• Compromise Power Analysis: When the respondent pool is fixed (hard-to-reach or elite samples), balance Type I and Type II error; an alpha above 0.05 can be defensible when it minimizes the combined error rate (Lakens 2025).
• Tools: cjpowR (Freitag 2021) or its Shiny app for simulation-based power curves for main effects and interactions. For a declare-diagnose-redesign workflow across estimands, diagnosands, and assignment schemes, use DeclareDesign (Blair, Cooper, Coppock, and Humphreys 2019).

4. Treatment Validation and Realism

• Experimental vs. Mundane Realism: Mundane realism (resemblance to real decisions) is neither necessary nor sufficient; what matters is experimental realism — whether the task creates the intended psychological state (Druckman 2022, citing Aronson and Carlsmith 1968). A tabular display can lack mundane realism yet succeed if respondents attend to and process the information.
• Attention and Salience: Even with internal validity secured, survey effects can amplify or reverse real-world effects because the survey compresses the consideration set (inflating AMCE magnitudes) and distorts relative attribute salience (which can flip signs) (Fu and Li 2024). Ask whether the table forces attention to attributes respondents would ignore in the real-world analog, and whether relative salience mirrors the target environment.
• Access, Attention, Processing: Validate that respondents (1) can see the attribute information, (2) read it, and (3) interpret it as intended. Attention checks and comprehension probes address 1–2; pilots and cognitive interviews address 3 (Druckman 2022).
• Names as Cues: Proper names, cultural referents, and country names can carry associations beyond the intended dimension. Pilot-test what respondents read into them.
• Practice Vignette: A non-experimental practice task before the conjoint block reduces learning effects in early tasks.
• Repeated Task for IRR: Plan to repeat the first task at the end of the block with left/right order reversed. This estimates intra-respondent reliability (about 77% agreement on average across Clayton et al.'s (2023) eight replications), respondents generally do not detect the repetition, and it enables bias-corrected AMCEs and marginal means via projoint. It costs one task and cannot be retrofitted after fielding.