mdhewei/bioinfor-claw/preprint-tracker

Preprint Tracker

Purpose

Monitor emerging research trends by searching recent preprints on bioRxiv (life sciences) and medRxiv (medical/clinical research). Identifies trending topics, most active authors, and generates formatted reports to stay ahead of published literature.

When to Use / When NOT to Use

Use when:

• You want to track the very latest research before peer review and formal publication
• Monitoring a field for rapid developments (e.g., new treatments, emerging pathogens)
• Identifying prolific authors and research groups in a specific area
• Building a curated digest of preprints on a topic for a journal club or group
• Finding cutting-edge methods before they appear in peer-reviewed journals

Do NOT use when:

• You need established, peer-reviewed evidence (use pubmed-search instead)
• You want to cite papers (preprints are not peer-reviewed; use published versions)
• You need papers older than 3-6 months (bioRxiv/medRxiv archive is limited)
• Your focus is on validating conclusions (preprints lack peer review filtering)

Expected Inputs & Outputs

Inputs:

• --query STRING — Search keywords (required; searches title and abstract)
• --server — Source: biorxiv, medrxiv, or both (default: both)
• --date-from STR — Start date (YYYY-MM-DD; default: 30 days ago)
• --date-to STR — End date (YYYY-MM-DD; default: today)
• --max-results INT — Maximum preprints to retrieve (default: 100)
• --category STR — bioRxiv category filter (e.g., "genomics", "bioinformatics", "cancer biology")
• --sort-by — Sort by: date (default, newest first) or relevance
• --output-format — Choices: tsv, markdown (default: markdown)

Outputs (in --outdir):

• preprints.tsv — Tabular results: doi, title, authors, category, date, word_count, abstract_preview
• trending_topics.tsv — Top keywords with frequency and top 3 papers using each keyword
• digest_report.md — Formatted preprint summaries with full metadata
• category_distribution.png — Bar chart of preprints by bioRxiv category
• timeline.png — Preprints published per week (bar chart)
• topic_frequency.png — Bar chart of top 20 keywords
• prolific_authors.tsv — Authors ranked by number of preprints
• search_log.txt — Processing notes and summary statistics

Procedure

1. Query bioRxiv/medRxiv API

• Call https://api.biorxiv.org/details/<server>/<date_from>/<date_to>/<cursor>/json
• Paginate through results (cursor: 0, 100, 200, etc.)
• Extract: doi, title, authors, category, abstract, date, jatsxml

1. Filter by Keywords

• Search title and abstract text for --query keywords (case-insensitive)
• Apply --category filter if provided

1. Extract Metadata

• Parse author list (first, last, affiliations where available)
• Count word count in abstract
• Parse publication date and category

1. Analyze Trending Topics

• Tokenize all abstracts and titles
• Remove stopwords
• Calculate word frequency
• Identify top 20 keywords

1. Identify Prolific Authors

• Count papers per author (both first author and all authors)
• Rank by publication count

1. Generate Outputs

• TSV: machine-readable tabular format
• Markdown: human-readable formatted preprint list
• Visualizations: publication timeline, category distribution, keyword frequency
• Author rankings for identifying key researchers

Key Execution Patterns

Track latest preprints in a topic:

python preprint_tracker.py --query "CRISPR cancer" --max-results 100 --outdir ./latest

Monitor bioRxiv only for computational methods:

python preprint_tracker.py --query "deep learning genomics" --server biorxiv \
  --category bioinformatics --outdir ./compbio

Track medical preprints from last week:

python preprint_tracker.py --query "treatment COVID-19" --server medrxiv \
  --date-from 2024-03-01 --date-to 2024-03-07 --outdir ./weekly

Find preprints from a specific timeframe:

python preprint_tracker.py --query "gene therapy" \
  --date-from 2024-01-01 --date-to 2024-03-31 --outdir ./q1_2024

Export as TSV for spreadsheet analysis:

python preprint_tracker.py --query "immunotherapy cancer" --output-format tsv \
  --max-results 200 --outdir ./analysis

Parameter Decision Guide

Failure Modes & Recovery

Special Notes

bioRxiv categories include: genomics, bioinformatics, cancer biology, cell biology, computational biology, synthetic biology, systems biology, evolutionary biology, genetics, immunology, microbiology, neuroscience, molecular biology, plant biology, and many others.

medRxiv categories include: epidemiology, clinical practice, infectious diseases, oncology, cardiology, psychiatry, and others.

API Rate Limits: bioRxiv/medRxiv APIs are generally unrestricted, but large batch requests (>500 results) may see delays.

Requirements

See requirements.txt for exact versions. Core dependencies:

• requests — HTTP API calls to bioRxiv/medRxiv
• pandas — Data organization and TSV export
• matplotlib — Visualization of trends and distributions
• json — Metadata serialization (stdlib)
• re — Regular expressions for text processing (stdlib)
• datetime — Date handling (stdlib)