draco-agent/tech-news-digest/tech-news-digest

Tech News Digest

Evidence-led tech digest with unified collection and one editorial selection shared across formats. Follow references/digest-prompt.md as the authoritative mode policy.

Editorial Contract

• Daily preserves original sections: 2–4 sentence overview; configured topics (normally LLM, AI Agent, Crypto, Frontier Tech; aim for 3–5 qualified items each); KOL viewpoints ≤3; releases ≤3 repositories; project discovery ≤3; blog picks ≤3; ≤30 unique items overall. News/blog explanations normally contain 2–3 concise sentences of change, evidence and implication. Retain empty headings with a coverage note, never filler. Optimize content within sections: a request to shorten GitHub releases must not become a whole-report redesign or one-message digest.
• Weekly ≤35 unique items: overview; configured topic sections in configured order (normally LLM, AI Agent, Crypto, Frontier Tech), with 2–4 substantial syntheses per topic as an advisory aim. Each synthesis uses 2–4 concise sentences joining week progress, evidence and implications, not concatenated headlines. Retain empty topic, key-release, project and viewpoint/reading headings with honest non-bullet coverage notes. Key releases ≤5 repositories (Chinese explanation ≤80 characters); projects ≤3; viewpoints/deep reading ≤4; optional next-week watch ≤3 pending releases, fixes or uncertainties. Do not force experiments; factual watch claims need citations, clearly proposed/not-performed experiments retain the narrow citation exemption.
• Caps are not quotas. Count evidence events inside syntheses and standalone watch items. Preserve topic sections in both modes; no global three-theme constraint. Body limits: daily ≤6500 characters, weekly ≤10000, excluding URL targets; never fill to the limit.
• Mandatory security coverage: follow the primary-alert/search and pre-delivery checks in references/digest-prompt.md. Preserve existing topics; use optional 🛡️ 安全警报 after the overview only for credible urgent incidents in either mode; weekly may also use an optional dedicated security/risk section for consequential nonurgent risks. Omit when empty and count/deduplicate within existing caps. Log checks and decisions; failed collection is not evidence of no incidents.
• Relevance, impact and evidence before internal scores. Exceptional claims need original technical evidence and independent assessment where possible, otherwise exclude/downgrade. No public scores, social metrics or operational statistics. Daily keeps substantive, nonduplicative KOL viewpoints. A material coverage gap may receive one concise caveat without raw counts; details stay in the final operational log.
• Deduplicate events and canonical URLs across the whole report. Daily repeats require incremental developments; weekly may synthesize daily coverage. Daily keeps blog picks; weekly viewpoints/deep reading may include original opinions, papers, docs, postmortems or essays.
• Project discovery is not verified trending; no lifetime-derived growth or repeated mature repos without meaningful changes. Preserve the prompt's consequential-release rules, exact versions and official links.
• Plain bullets with bold titles; concise inline Discord links [来源](<URL>). Section-aware chunks ≤1700 characters including numbering. Apply the requested language, including Simplified Chinese explanations when configured.
• Validate the canonical archive with python3 scripts/validate-digest.py --input FILE --mode daily or --mode weekly before delivery; fix all errors. Weekly production runs append one --topic-heading 'EXACT LOCALIZED HEADING' per configured topic in order; without these flags the gate cannot infer custom headings. Missing/failing validation blocks delivery. Manually verify unique-event counts inside syntheses, required non-topic headings/overview, security placement, evidence/coverage honesty and cross-format parity too.
• Discord, Markdown, email and PDF use identical selections/judgments. Operational stats/runtime attribution belong in the final log, not content.
• Legacy ITEMS_PER_SECTION, BLOG_PICKS_COUNT and EXTRA_SECTIONS are advisory parameters superseded by mode policy: ignore their counts/extra sections, especially duplicate weekly trends.

Quick Start

1. Configuration Setup: Default configs are in config/defaults/. Copy to workspace for customization:

``bash mkdir -p workspace/config cp config/defaults/sources.json workspace/config/tech-news-digest-sources.json cp config/defaults/topics.json workspace/config/tech-news-digest-topics.json ``

1. Environment Variables:

• TWITTERAPI_IO_KEY - twitterapi.io API key (optional, preferred)
• X_BEARER_TOKEN - Twitter/X official API bearer token (optional, fallback)
• TAVILY_API_KEY - Tavily Search API key, alternative to Brave (optional)
• WEB_SEARCH_BACKEND - Web search backend: auto|brave|tavily (optional, default: auto)
• BRAVE_API_KEYS - Brave Search API keys, comma-separated for rotation (optional)
• BRAVE_API_KEY - Single Brave key fallback (optional)
• GITHUB_TOKEN - GitHub personal access token (optional, improves rate limits)

1. Generate Digest:

``bash # Unified pipeline (recommended) — runs all 6 sources in parallel + merge python3 scripts/run-pipeline.py \ --defaults config/defaults \ --config workspace/config \ --hours 48 --freshness pd \ --archive-dir workspace/archive/tech-news-digest/ \ --output /tmp/td-merged.json --verbose --force ``

1. Select and Validate: Follow references/digest-prompt.md; validate the canonical Markdown before rendering the same selections to Discord, email and PDF.

Configuration Files

sources.json - Unified Data Sources

{
  "sources": [
    {
      "id": "openai-rss",
      "type": "rss",
      "name": "OpenAI Blog",
      "url": "https://openai.com/blog/rss.xml",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "ai-agent"],
      "note": "Official OpenAI updates"
    },
    {
      "id": "sama-twitter",
      "type": "twitter", 
      "name": "Sam Altman",
      "handle": "sama",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "frontier-tech"],
      "note": "OpenAI CEO"
    }
  ]
}

topics.json - Enhanced Topic Definitions

{
  "topics": [
    {
      "id": "llm",
      "emoji": "🧠",
      "label": "LLM / Large Models",
      "description": "Large Language Models, foundation models, breakthroughs",
      "search": {
        "queries": ["LLM latest news", "large language model breakthroughs"],
        "must_include": ["LLM", "large language model", "foundation model"],
        "exclude": ["tutorial", "beginner guide"]
      },
      "display": {
        "max_items": 8,
        "style": "detailed"
      }
    }
  ]
}

Scripts Pipeline

run-pipeline.py - Unified Pipeline (Recommended)

python3 scripts/run-pipeline.py \
  --defaults config/defaults [--config CONFIG_DIR] \
  --hours 48 --freshness pd \
  --archive-dir workspace/archive/tech-news-digest/ \
  --output /tmp/td-merged.json --verbose --force

• Features: Runs all 6 fetch steps in parallel, then merges + deduplicates + scores
• Output: Final merged JSON ready for report generation (~30s total)
• Metadata: Saves per-step timing and counts to *.meta.json
• GitHub Auth: Auto-generates GitHub App token if $GITHUB_TOKEN not set
• Fallback: If this fails, run individual scripts below

Individual Scripts (Fallback)

fetch-rss.py - RSS Feed Fetcher

python3 scripts/fetch-rss.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--verbose]

• Parallel fetching (10 workers, max 2 concurrent per host), retry with jittered backoff, feedparser + regex fallback
• Timeout: 30s per feed, ETag/Last-Modified caching in $TECH_NEWS_DIGEST_STATE_DIR

fetch-twitter.py - Twitter/X KOL Monitor

python3 scripts/fetch-twitter.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--backend auto|official|twitterapiio]

• Backend auto-detection: uses twitterapi.io if TWITTERAPI_IO_KEY set, else official X API v2 if X_BEARER_TOKEN set
• Rate limit handling, engagement metrics, retry with backoff

fetch-web.py - Web Search Engine

python3 scripts/fetch-web.py [--defaults DIR] [--config DIR] [--freshness pd] [--output FILE]

• Auto-detects Brave API rate limit: paid plans → parallel queries, free → sequential
• Without API: generates search interface for agents

fetch-github.py - GitHub Releases Monitor

python3 scripts/fetch-github.py [--defaults DIR] [--config DIR] [--hours 168] [--output FILE]

• Parallel fetching (10 workers), 30s timeout
• Auth priority: $GITHUB_TOKEN → GitHub App auto-generate → gh CLI → releases.atom
• Without a token (or on a 403/429 rate limit) it reads github.com/<repo>/releases.atom, which needs no auth,

consumes no REST quota, and also surfaces tag-only repos such as torvalds/linux

• The Atom feed lists every tag, so entries are kept only when the tag carries a <major>.<minor>

version (v2.13.0, v6.19-rc4). This drops CI tags such as PyTorch's trunk/<sha> and viable/strict/<epoch>, which would otherwise crowd out real releases

fetch-github.py --trending - Project Discovery (legacy flag)

python3 scripts/fetch-github.py --trending [--hours 48] [--output FILE] [--verbose]

• Searches GitHub API for discovery candidates; search results do not establish genuine trends.
• Neutral internal base score 5, not a growth signal. No daily_stars_est or lifetime-derived growth. Daily discovery ≤3, weekly ≤3; require a current reason to try the project.

fetch-reddit.py - Reddit Posts Fetcher

python3 scripts/fetch-reddit.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE]

• Parallel fetching (4 workers), public JSON API (no auth required)
• 13 subreddits with score filtering

enrich-articles.py - Article Full-Text Enrichment

python3 scripts/enrich-articles.py --input merged.json --output enriched.json [--min-score 10] [--max-articles 15] [--verbose]

• Fetches full article text for high-scoring articles
• Cloudflare Markdown for Agents (preferred) → HTML extraction (fallback) → Skip (paywalled/social)
• Blog domain whitelist with lower score threshold (≥3)
• Parallel fetching (5 workers, 10s timeout)

merge-sources.py - Quality Scoring & Deduplication

python3 scripts/merge-sources.py --rss FILE --twitter FILE --web FILE --github FILE --reddit FILE

• Quality scoring, URL dedup (tracking params stripped, query preserved), title similarity dedup (75%), previous digest penalty
• Multi-source bonus: +5 per additional source type corroborating the same story
• Topic priority is derived from topics.json order; pass --defaults/--config to keep it in sync
• Output: topic-grouped articles sorted by score

validate-config.py - Configuration Validator

python3 scripts/validate-config.py [--defaults DIR] [--config DIR] [--verbose]

• JSON schema validation, topic reference checks, duplicate ID detection

generate-pdf.py - PDF Report Generator

python3 scripts/generate-pdf.py --input report.md --output digest.pdf [--verbose]

• Converts markdown digest to styled A4 PDF with Chinese typography (Noto Sans CJK SC)
• Emoji icons, page headers/footers, blue accent theme. Requires weasyprint.

sanitize-html.py - Safe HTML Email Converter

python3 scripts/sanitize-html.py --input report.md --output email.html [--verbose]

• Converts markdown to XSS-safe HTML email with inline CSS
• URL whitelist (http/https only), HTML-escaped text content