menkesu/awesome-pm-skills/ai-product-bets

AI Product Bets

[Image: ai-product-bets: lead guest team]

Decide which AI features to ship now, build for the next model, shelve with a retest trigger, or kill, and prove each one gets better as models improve. Built from 405 insights from 118 Lenny's Podcast guests. Lead team: Nick Turley (ChatGPT), Kevin Weil (OpenAI), Dianne Penn (Anthropic).

When to use

• You have a list of AI feature ideas (or a roadmap/PRD) and need to cut it to the few that matter
• A new model just shipped and you need to know what it unlocks, obsoletes or makes cheaper
• Someone says 'we need an AI strategy' and the board or CEO wants a point of view this quarter
• A feature underperformed or 'didn't work yet' and you must decide to kill or shelve it
• You are choosing agent vs assistant vs chat, or frontier vs cheaper vs custom model
• You fear a lab (OpenAI, Anthropic, Google) will ship your product as a feature
• You are an incumbent deciding between bolting AI on and rebuilding around it
• You are an enterprise buyer or builder planning to automate a critical process

Step 1 — Diagnose (ask before answering)

If the user attached a roadmap, PRD, deck or repo, read it first, extract the feature list, and only ask what is still missing. Max 4 questions:

1. Where do you sit? Model builder / app on someone else's model / incumbent adding AI / enterprise automating a process. Routes to: embed-with-research vs scaffolding ledger vs rebuild-or-bolt-on vs process-automation play.
2. What does a wrong answer cost, and what is the hit rate today? (Rough % on real tasks.) Routes to: model maximalism vs scaffold-what-you-cannot-tolerate; UI sizing.
3. Can you survive 3-6 months of weak traction waiting for a model? Routes to: edge-of-capability bet vs ship-what-works-now.
4. What proprietary data, distribution or workflow do you own, and which candidate features already failed? Routes to: moat test and the retest shelf.

Step 2 — Pick the play

Step 3 — Run it

Play A — Bet triage (always run first)

1. List every candidate feature (cap at 15). Write the user job in one line each. Start from what the product does and why people use it (Paul Adams), not from the technology.
2. Problem-first gate. Delete any feature without a named user, pain and baseline. Ask: is this problem real without the word AI in it? (Inbal Shani, Marily Nika, Melanie Perkins.) Chatbot bolted on to an existing product = automatic fail (Cameron Adams, Paul Adams).
3. Baseline accuracy check (Jason Droege). Human process 10-20% accurate or liked: AI wins big, route the remainder to humans. Human process already ~98%: do not expect AI to deliver the last 2%.
4. 2X test (Nick Turley). If the model were twice as smart tomorrow, would this feature be twice as good? Tag each feature:

• Rides the curve: invest.
• Crutch: patches a model weakness (Cat Wu's to-do list). Time-box it and write the deletion trigger.
• Orthogonal: SOC 2, permissions, compliance. Required, but do not count it as an AI bet.
• Not yet: works under ~50% of the time. Goes to Play B.

1. Claude 8 test (Dianne Penn). Write 3 sentences: what do users do differently when a much stronger model arrives, and does today's UX survive it? Features that disappear in that world are crutches.
2. Feature or category? (Marc Andreessen.) Is AI an added ingredient (AI editing in Photoshop) or does it make the workflow unnecessary (generate instead of edit)? Category answer = rebuild, not retrofit. Also ask the hard-generative test (Gustav Soderstrom): could this product exist without generative AI? Most things you list will be iterative improvements; budget them as such.
3. Verdict per feature: Ship now / Build for next model / Shelve + retest / Kill. Done when every feature has one verdict and one owner.

Play B — Retest shelf (build what does not work yet)

1. For each Not-yet feature, prototype for one week against the current frontier model (Andrew Ambrosino: prototype all of them, ship what is ready, let the rest bake).
2. Write a capability-gap statement: the specific failure (misses 1 of 20 call sites, invents policy) not 'quality is low' (Cat Wu: know what is missing for it to work).
3. Keep the prototype and failing cases as fixtures. Failure in market no longer proves the shape is wrong; Operator, Atlas agent and Codex are one idea re-released as intelligence improved (Ambrosino).
4. Set the retest trigger: every new model release (roughly every 3-4 months, Kevin Weil). Run the fixtures first, the same week the model lands (Eric Simons shipped Bolt after a model crossed the threshold; Dharmesh Shah revived a failed bot six years later).
5. Match form factor to today's ability. Original Codex web delegated whole tasks and was too early; local, ask-questions Claude Code fit what models could do then (Ambrosino). Ship the shape that works at today's hit rate; plan the next generation (Scott Wu: expect many generations of agent product).
6. Check product overhang: what today's model already does that users do not know (Dianne Penn). Surface it before building more.

Play C — Scaffolding ledger + model-release ritual

1. Keep a table: scaffold | weakness it patches | measured gain | cost to keep | delete trigger.
2. Build rule. Expected gain 10-20% and a model release is likely within 6 months: do not build, wait (Boris Cherny). Keep scaffolding only for errors you can never tolerate (Kevin Weil).
3. Every model launch: read the entire system prompt and ask per section whether the model still needs the reminder; remove it if not (Cat Wu). De-emphasize crutch features in the UI.
4. Pull forward, then delete (Roman Ugarte): predict what the next model makes baseline, build it now with engineering, delete the scaffold when it lands, move to the next impossible thing.
5. Model sommelier habit (Anish Acharya): ship something small with every new model; log its shape (creative, precise, long-horizon, vision). Models are not fungible.
6. Update priors (Aparna Chennapragada): re-run anything that failed a few months ago before you say 'AI cannot do that'.

Play D — Reliability bar and fault-tolerant design

1. Measure the hit rate on 50-100 real tasks (see eval-plan). Without it, every other decision is a guess.
2. Size the UI to the hit rate (Gustav Soderstrom): near-zero error needed for one big button; at 1-in-5 hit rate show about five candidates plus an easy 'no, wrong' escape hatch. Midjourney's four-image grid is the canonical case.
3. A 30% acceptance rate is not annoying if users learn which suggestions to ignore and mistakes are cheap to dismiss (Varun Mohan, Alexander Embiricos). It is fatal when a wrong output hurts the user (Alex Komoroske: design so failure is not catastrophic).
4. Budget the nines. POCs hit 60-70% fast; each extra nine of reliability costs about an order of magnitude more (Droege). Plan 6-12 months for a robust important enterprise process, 4-6 months to real ROI on a critical workflow (Aishwarya Naresh Reganti). One-click agents are marketing.
5. Make the UI promise match data quality; show sources and uncertainty (Noah Weiss). Ask the jagged-frontier questions for non-experts (Benedict Evans): can the user tell where it will work, tell after it worked, and decide what to do with it?
6. For algorithmic products, assign each decision to the algorithm or to a person and give operators controls (Adriel Frederick). ML optimizes inside constraints humans set.

Play E — Moat and platform-risk test

1. Layer. Frontier models need $2B+ raisable without product progress (Ben Horowitz, Bret Taylor): not a startup play. Tooling is viable if you can answer why customers will not take the infra provider's version. Applied agents for one deeply understood business problem is the open lane.
2. Shovels, mine or middle (Matt MacInnis): sell the model, own proprietary data, or you are in the middle renting both and need a durable insight.
3. Lab test (Logan Kilpatrick): general assistants and agents will be built by labs; verticalized products (sales agent, legal) will not. Ask: would a lab ship this inside its general product within 12 months?
4. Seven powers lens (Hamilton Helmer): AI adds no eighth power. Ask whether fixed model costs spread across users (scale), whether one user's usage improves another's (network), whether the model learns a user in a way that raises switching costs.
5. Differentiator must improve as LLMs improve (Tamar Yehoshua). If your edge is compensating for model weakness it expires. Context and memory is the platform moat (Brian Balfour); data freshness and structure is about 90% of the effort (Shaun Clowes).
6. Build / partner / adapt (Cameron Adams, Michael Truell, Asha Sharma): partner for commodity capability, build where you have a data advantage, adapt an existing model with your data rather than training from scratch. Custom models only where cost or speed forces it (autocomplete under 300 ms) or foundation models are weak.
7. Economics. Capability cost falls about 10x a year (Kevin Weil), but inference is a real cost today: capture value from day one (Madhavan Ramanujam) and re-run unit economics at each release.