9600dev/mmr/news

News Skill (mmr)

LLMVM: Use <helpers>...</helpers> blocks to call NewsHelpers methods. All methods are async. Returns are dicts in the mmr convention — check result.get("ok") and report fetcher_source before trusting content.

This skill wraps a local Dockerized scraper that lives in ~/dev/scraper (renamed from ~/dev/news in mid-2026; config is ~/.config/scraper/scraper.yaml). The service provides browser-style fetching (Playwright + stealth), an httpx → playwright → scrapingbee → archive.ph fallback chain, aggressive HTML→Markdown extraction via rustdown, a persistent searchable scrape store (use_cache, GET /v1/scrapes), server-side image handling (images: urls|hosted|embed), and (optionally) LLM extraction over the scraped body. The generic, always-current description of the service is the scraper repo's own skill at ~/dev/scraper/scraper-service-skill/; the reference/ docs here are copied from it. What this skill adds is the mmr composition layer (ticker_news, portfolio_news, the freshness recipes, FINANCIAL_SOURCES.md).

When to invoke this skill

Trigger when the user:

• Asks you to fetch / scrape / read / summarize a specific article URL
• Asks for news on a ticker or portfolio position (use ticker_news() / portfolio_news())
• Wants a narrative digest on a market catalyst (use search_and_scrape())
• Hits a paywall, 403, or Cloudflare challenge on a finance source
• Wants Reuters / Bloomberg / WSJ / FT / CNBC / Seeking Alpha / Stockhead / Motley Fool AU / Market Index article content

Do not invoke for:

• Raw price/quote lookups — use mmr-skill (MMRHelpers.snapshot, MMRHelpers.history_massive)
• Fundamentals — use mmr-skill (MMRHelpers.balance_sheet, etc.)
• Arbitrary non-article URLs (API endpoints, images, GitHub pages)

Service pre-flight: always check health first

h = await NewsHelpers.health()
# {"status": "ok", "version": "0.1.0", "llm_enabled": true, "llm_verified": true,
#  "mathpix_enabled": true, "scrapingbee_enabled": true, "store_enabled": true,
#  "agent_enabled": true, "scrapingbee_usage": {"calls_24h": 0, "credits_24h": 0, ...},
#  "degraded_reasons": [], "egress_filter": "enforced"}

status is "ok" only when the core pipeline is up AND no enabled optional subsystem is failing; a non-empty degraded_reasons names the subsystem and its last error. scrapingbee_usage is the paid-tier spend — glance at it before a batch run.

If this returns ok: False with "connection refused", the service isn't running. Start it:

cd ~/dev/scraper && ./docker.sh -g

-g builds (if needed), starts, and tails logs. Wait for scraper service ready on http://... in the log, then re-check health.

NewsHelpers — Method Summary

Core patterns

Pattern 1: Read a specific article

res = await NewsHelpers.scrape("https://www.reuters.com/markets/...")
if res.get("ok") and res.get("status") == "ok":
    art = res["article"]
    print(art["title"])
    print(f"fetched via: {art['fetcher_source']}")
    print(f"published: {art.get('metadata', {}).get('published_at')}")
    print(art["markdown"])

Always inspect `fetcher_source` — if it starts with archive: the content is a snapshot and may be days/weeks/months old. That matters for trading decisions. Mention it to the user.

Pattern 2: News on a specific ticker

res = await NewsHelpers.ticker_news("BHP", max_results=5, exchange_hint="ASX")
for a in res["articles"]:
    if a.get("ok"):
        art = a["article"]
        meta = a.get("search", {})
        print(f"[{meta.get('source')}] {art['title']}")
        print(f"  {meta.get('published_at')} · via {art['fetcher_source']}")

exchange_hint disambiguates foreign tickers. Use "ASX", "LSE", "HKEX", "TSE" where relevant. Without it, the query is just "{TICKER} stock news" which biases US-centric.

Pattern 3: News digest for an entire portfolio

portfolio = await MMRHelpers.portfolio()
tickers = [p["symbol"] for p in portfolio["data"][:10]]
news = await NewsHelpers.portfolio_news(tickers, max_results_per_ticker=3)
for sym, articles in news["by_ticker"].items():
    print(f"\n=== {sym} ===")
    for a in articles:
        if a.get("ok"):
            print(f"  - {a['article']['title']}")

Pattern 4: Narrative digest on a macro catalyst

bundle = await NewsHelpers.search_and_scrape(
    "Fed rate decision 2026 dot plot",
    max_results=5,
    engine="google_news",
)
# Hand the lot to llm_call for synthesis
corpus = "\n\n---\n\n".join(
    a["article"]["markdown"] for a in bundle["articles"] if a.get("ok")
)
digest = await llm_call([corpus], "Summarize the narrative and quote key voices.")
emit(digest)

Pattern 5: Targeted LLM extraction (requires llm_enabled: true on the service)

res = await NewsHelpers.scrape(
    "https://www.reuters.com/...",
    extract="Name the CEO quoted and summarize their statement in one sentence.",
)
if res.get("extraction"):
    print(res["extraction"]["text"])
# If the service has llm_enabled=false this returns HTTP 422 with
# error.code == "llm_disabled" — see reference/operations.md to enable.

Pattern 6: Follow-link synthesis (mode 3)

res = await NewsHelpers.scrape(
    "https://www.reuters.com/markets/weekly-wrap/",
    extract="Which central banks moved rates this week and by how much?",
    follow_links=3,
)
# The LLM picks up to 3 linked URLs from the seed, scrapes them in parallel,
# and synthesises the extraction across the combined corpus.
print(res.get("extraction", {}).get("text"))
print("followed:", [p["url"] for p in res.get("follow_pages", []) if p.get("ok")])

Pattern 7: Extract images from an article (LLM-visible)

The news-service strips images during HTML→Markdown conversion. Use get_images() when the visual content is part of the story — a chart, a price graph, a map, a product photo, a CEO headshot.

res = await NewsHelpers.get_images(
    "https://stockhead.com.au/resources/...",
    max_images=3,       # cap — LLM visual budget is not infinite
    min_bytes=5000,     # filters out logos, icons, tracking pixels
)
if res["ok"]:
    for img in res["images"]:
        emit(img)                # LLM can now see the image
    for m in res["image_meta"]:
        print(f"  {m['alt']} ({m['bytes']} bytes, {m['content_type']})")

Under the hood this:

1. Calls scrape(url, include_html=True) to get the cleaned HTML.
2. Parses <img> tags (including lazy-load variants data-src, data-lazy-src).
3. Resolves relative URLs, filters ad/analytics hosts, SVGs, GIFs.
4. Downloads surviving candidates in parallel via httpx.
5. Drops anything under min_bytes (most logos/icons/favicons are <5KB).
6. Wraps remaining bytes as llmvm_lite.llm_types.ImageContent objects.

Returns:

{
  "ok": True,
  "article_title": "…",
  "article_url": "…",              # final URL after redirects
  "fetcher_source": "httpx",        # or "playwright", "archive:*"
  "count": 3,
  "images": [ImageContent, ...],    # pass to emit() or llm_call()
  "image_meta": [{src, alt, bytes, content_type}, ...],
  "skipped": [{src, reason}, ...],  # why each candidate was dropped
}

Use sparingly. Each image adds meaningful tokens to the LLM context (a 1200×675 JPEG is typically 1,500–3,000 visual tokens). 1–3 images per article is usually the sweet spot for market commentary; set max_images=1 if you only need the hero image.

Works via the scrape fallback chain. Since get_images() piggybacks on scrape(include_html=True), it benefits from the same httpx → playwright → archive tier — so it'll still pull images out of Cloudflare-protected or paywalled articles that come through the archive tier.

Passing to `llm_call()` for visual analysis:

res = await NewsHelpers.get_images(chart_article_url, max_images=1)
if res["images"]:
    analysis = await llm_call(
        res["images"] + [res.get("markdown", "")],
        "Read the chart in the image and tell me the key price levels.",
    )
    emit(analysis)

Finding the latest financial news

The server doesn't support date filters directly — fresh news is a function of (a) using `google_news` (only engine that returns published_at), (b) phrasing queries with temporal anchors ("today", "this week", the current month/year), and (c) sorting/filtering on `published_at` client-side after fetching.

Recipe A: latest news for a single ticker

ticker_news() already uses google_news under the hood. For max recency, add a date anchor to the query by passing it through search_and_scrape():

import datetime as dt
this_month = dt.datetime.now().strftime("%B %Y")  # e.g. "April 2026"
bundle = await NewsHelpers.search_and_scrape(
    f"AGE Alligator Energy ASX news {this_month}",
    max_results=8,
    engine="google_news",
)
# Sort by published_at (most recent first); None values sink to the bottom.
articles = sorted(
    [a for a in bundle["articles"] if a.get("ok")],
    key=lambda a: a.get("search", {}).get("published_at") or "",
    reverse=True,
)
for a in articles[:5]:
    s = a["search"]
    print(f"{s.get('published_at')} · [{s.get('source')}] {a['article']['title']}")

Recipe B: latest news on an entire sector

For "what's moving in ASX uranium miners this week" style scans: