lightning-rod-labs/lightningrod-python-sdk/verify-forecasting-docs

Verify Forecasting Docs

Two-phase verification of docs/forecasting/: run every code snippet against the live API, then check that the prose around each snippet matches observed behavior.

Setup

Ask the user for an API key if one hasn't been provided. The project .env may point to staging — bypass it so snippets run against production:

LIGHTNINGROD_BASE_URL="https://api.lightningrod.ai/v1" python ...

Activate the venv before running any Python:

source venv/bin/activate

Phase 1: Execute every code snippet

Read each doc file in turn. For every fenced Python block, run it verbatim (substituting the real API key for "your-api-key"). Capture stdout and any exception.

Cover all files under docs/forecasting/** in the order they are listed in SUMMARY.md.

For each snippet record:

• file and approximate line range
• exit status (pass / error / exception type + message)
• stdout (first 300 chars is enough)

Phase 2: Semantic validation

For each snippet that passed, check the prose immediately before and after it against what you actually observed. Flag any mismatch as a finding.

Common things to check:

Flag findings as:

• ⚠️ wrong — prose makes a claim that contradicts observed behavior
• ⚠️ stale — prose describes something that no longer exists (removed field, renamed param)
• 📝 missing — behavior exists but isn't documented

Report format

## Doc Verification: forecasting docs
 
**Verdict:** PASS | FAIL
 
### Snippet results
| # | File | Description | Result |
|---|------|-------------|--------|
| 1 | quickstart.md | OpenAI client minimal | ✅ pass |
| 2 | sdk.md | lr.predict() minimal | ✅ pass |
...
 
### Semantic findings
- ⚠️ <finding>
- 📝 <finding>
(empty = all prose checked out)

FAIL if any snippet raises an unexpected exception, or if any ⚠️ finding is found. 📝 findings don't fail but should be noted.