drchronx/ai-agent-research-starter-kit/evaluating-paper-relevance

Evaluating Paper Relevance

Overview

Two-stage screening process: quick abstract scoring followed by deep dive into promising papers.

Core principle: Precision over breadth. Find papers that actually contain the specific data/methods user needs, not just topically related papers.

When to Use

Use this skill when:

• Have list of papers from search
• Need to determine which papers have relevant data
• User asks for specific information (measurements, protocols, datasets, etc.)
• Screening papers one-by-one
• Any research domain (medicinal chemistry, genomics, ecology, computational methods, etc.)

Choosing Your Approach

Small searches (<50 papers):

• Manual screening with progress reporting
• Use papers-reviewed.json + SUMMARY.md only
• No helper scripts needed
• Report progress to user for every paper

Large searches (50-150 papers):

• Consider helper scripts (screen_papers.py + deep_dive_papers.py)
• Use Progressive Enhancement Pattern (see Helper Scripts section)
• Create README.md with methodology
• May want TOP_PRIORITY_PAPERS.md for quick reference
• Use richer JSON structure (evaluated-papers.json categorized by relevance)
• Consider using subagent-driven-review skill for parallel screening

Very large searches (>150 papers):

• Definitely use helper scripts with Progressive Enhancement Pattern
• Create full auxiliary documentation suite (README.md, TOP_PRIORITY_PAPERS.md)
• Consider citation network analysis
• Plan for multi-week timeline
• Strongly consider subagent-driven-review skill for parallelization
• May need multiple consolidation checkpoints

Two-Stage Process

Stage 1: Abstract Screening (Fast)

Goal: Quickly identify promising papers

Score 0-10 based on:

• Keywords match (0-3 points): Does abstract mention key terms relevant to the query?
• Data type match (0-4 points): Does it mention the specific information user needs?
• Examples: measurements (IC50, expression levels, population sizes), protocols, datasets, structures, sequences, code
• Specificity (0-3 points): Is it specific to user's question or just general background/review?

Decision rules:

• Score < 5: Skip (not relevant)
• Score 5-6: Note in summary as "possibly relevant" but skip for now
• Score ≥ 7: Proceed to Stage 2 (deep dive)

IMPORTANT: Report to user for EVERY paper:

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 8 → Fetching full text...

or

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 4 → Skipping (insufficient relevance)

Never screen silently - user needs to see progress happening

Stage 2: Deep Dive (Thorough)

Goal: Extract specific data/methods from promising papers

1. Check ChEMBL (for medicinal chemistry papers)

If paper describes medicinal chemistry / SAR data:

Use skills/research/checking-chembl to check if paper is in ChEMBL database:

curl -s "https://www.ebi.ac.uk/chembl/api/data/document.json?doi=$doi"

If found in ChEMBL:

• Note ChEMBL ID and activity count in SUMMARY.md
• Report to user: "✓ ChEMBL: CHEMBL3870308 (45 data points)"
• Structured SAR data available without PDF parsing

Continue to full text fetch for context, methods, discussion.

2. Fetch Full Text

Try in order:

A. PubMed Central (free full text):

# Check if available in PMC
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=PMID[PMID]&retmode=json"
 
# If found, fetch full text XML via API
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMCID&rettype=full&retmode=xml"
 
# Or fetch HTML directly (note: use pmc.ncbi.nlm.nih.gov, not www.ncbi.nlm.nih.gov/pmc)
curl "https://pmc.ncbi.nlm.nih.gov/articles/PMCID/"

B. DOI resolution:

# Try publisher link
curl -L "https://doi.org/10.1234/example.2023"
# May hit paywall - check response

C. Unpaywall (MANDATORY if paywalled): CRITICAL: If step B hits a paywall, you MUST immediately try Unpaywall before giving up.

Use skills/research/finding-open-access-papers to find free OA version:

curl "https://api.unpaywall.org/v2/DOI?email=USER_EMAIL"
# Often finds versions in repositories, preprint servers, author copies
# IMPORTANT: Ask user for their email if not already provided - do NOT use claude@anthropic.com

Report to user:

⚠️  Paper behind paywall, checking Unpaywall...
✓ Found open access version at [repository/preprint server]

or

⚠️  Paper behind paywall, checking Unpaywall...
✗ No open access version available - continuing with abstract only

D. Preprints (direct):

• Check bioRxiv: https://www.biorxiv.org/content/10.1101/{doi}
• Check arXiv (for computational papers)

If full text unavailable AFTER trying Unpaywall:

• Note in SUMMARY.md: "⚠️ Full text behind paywall - no OA version found via Unpaywall"
• Continue with abstract-only evaluation (limited)

CRITICAL: Do NOT skip Unpaywall check. Many paywalled papers have free versions in repositories.

2. Scan for Relevant Content

Focus on sections:

• Methods: Experimental procedures, protocols
• Results: Data tables, figures, measurements
• Tables/Figures: Often contain the specific data user needs
• Supplementary Information: Additional data, extended methods

What to look for (adapt to research domain):

• Specific data user requested
• Medicinal chemistry: IC50 values, compound structures, SAR data
• Genomics: Gene expression levels, sequences, variant data
• Ecology: Population measurements, species counts, environmental parameters
• Computational: Algorithms, code availability, performance benchmarks
• Clinical: Patient outcomes, treatment protocols, sample sizes
• Methods/protocols described in detail
• Statistical analysis and significance
• Data availability statements
• Code/data repositories mentioned

Use grep/text search (adapt search terms):

# Examples for different domains
grep -i "IC50\|Ki\|MIC" paper.xml                    # Medicinal chemistry
grep -i "expression\|FPKM\|RNA-seq" paper.xml        # Genomics
grep -i "abundance\|population\|sampling" paper.xml  # Ecology
grep -i "algorithm\|github\|code" paper.xml          # Computational

3. Extract Findings

Create structured extraction (adapt to research domain):

Example 1: Medicinal chemistry

{
  "doi": "10.1234/medchem.2023",
  "title": "Novel kinase inhibitors...",
  "relevance_score": 9,
  "findings": {
    "data_found": [
      "IC50 values for compounds 1-12 (Table 2)",
      "Selectivity data (Figure 3)",
      "Synthesis route (Scheme 1)"
    ],
    "key_results": [
      "Compound 7: IC50 = 12 nM",
      "10-step synthesis, 34% yield"
    ]
  }
}

Example 2: Genomics

{
  "doi": "10.1234/genomics.2023",
  "title": "Gene expression in disease...",
  "relevance_score": 8,
  "findings": {
    "data_found": [
      "RNA-seq data for 50 samples (GEO: GSE12345)",
      "Differential expression results (Table 1)",
      "Gene set enrichment analysis (Figure 4)"
    ],
    "key_results": [
      "123 genes upregulated (FDR < 0.05)",
      "Pathway enrichment: immune response"
    ]
  }
}

Example 3: Computational methods

{
  "doi": "10.1234/compbio.2023",
  "title": "Novel alignment algorithm...",
  "relevance_score": 9,
  "findings": {
    "data_found": [
      "Algorithm pseudocode (Methods)",
      "Code repository (github.com/user/tool)",
      "Benchmark results (Table 2)"
    ],
    "key_results": [
      "10x faster than BLAST",
      "98% accuracy on test dataset"
    ]
  }
}

4. Download Materials

PDFs:

# If PDF available
curl -L -o "papers/$(echo $doi | tr '/' '_').pdf" "https://doi.org/$doi"

Supplementary data:

# Download SI files if URLs found
curl -o "papers/${doi}_supp.zip" "https://publisher.com/supp/file.zip"

5. Update Tracking Files

CRITICAL: Use ONLY papers-reviewed.json and SUMMARY.md. Do NOT create custom tracking files.

CRITICAL: Add EVERY paper to papers-reviewed.json, regardless of score. This prevents re-reviewing papers and tracks complete search history.

Add to papers-reviewed.json:

For relevant papers (score ≥7):

{
  "10.1234/example.2023": {
    "pmid": "12345678",
    "status": "relevant",
    "score": 9,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T10:30:00Z",
    "found_data": ["IC50 values", "synthesis methods"],
    "has_full_text": true,
    "chembl_id": "CHEMBL1234567"
  }
}

For not-relevant papers (score <7):

{
  "10.1234/another.2023": {
    "pmid": "12345679",
    "status": "not_relevant",
    "score": 4,
    "source": "pubmed_search",
    "timestamp": "2025-10-11T10:31:00Z",
    "reason": "no activity data, review paper"
  }
}

Always add papers even if skipped - this prevents re-processing and documents what was already checked.

Add to SUMMARY.md (examples for different domains):

Medicinal chemistry example:

### [Novel kinase inhibitors with improved selectivity](https://doi.org/10.1234/medchem.2023) (Score: 9)
 
**DOI:** [10.1234/medchem.2023](https://doi.org/10.1234/medchem.2023)
**PMID:** [12345678](https://pubmed.ncbi.nlm.nih.gov/12345678/)
**ChEMBL:** [CHEMBL1234567](https://www.ebi.ac.uk/chembl/document_report_card/CHEMBL1234567/)
 
**Key Findings:**
-IC50 values for 12 inhibitors (Table 2)
-Compound 7: IC50 = 12 nM, >80-fold selectivity
-Synthesis route (Scheme 1, page 4)
 
**Files:** PDF, supplementary data

Genomics example:

### [Transcriptomic analysis of disease progression](https://doi.org/10.1234/genomics.2023) (Score: 8)
 
**DOI:** [10.1234/genomics.2023](https://doi.org/10.1234/genomics.2023)
**PMID:** [23456789](https://pubmed.ncbi.nlm.nih.gov/23456789/)
**Data:** [GEO: GSE12345](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE12345)
 
**Key Findings:**
-RNA-seq data: 50 samples, 3 conditions
-123 differentially expressed genes (FDR < 0.05)
-Immune pathway enrichment (Figure 3)
 
**Files:** PDF, supplementary tables with gene lists

Computational methods example:

### [Fast sequence alignment with novel algorithm](https://doi.org/10.1234/compbio.2023) (Score: 9)
 
**DOI:** [10.1234/compbio.2023](https://doi.org/10.1234/compbio.2023)
**Code:** [github.com/user/tool](https://github.com/user/tool)
 
**Key Findings:**
-New alignment algorithm (pseudocode in Methods)
-10x faster than BLAST, 98% accuracy
-Benchmark datasets available
 
**Files:** PDF, code repository linked

IMPORTANT: Always make DOIs and PMIDs clickable links:

• DOI format: [10.1234/example.2023](https://doi.org/10.1234/example.2023)
• PMID format: [12345678](https://pubmed.ncbi.nlm.nih.gov/12345678/)
• Makes papers easy to access directly from SUMMARY.md

Progress Reporting

CRITICAL: Report to user as you work - never work silently!

For every paper, report:

1. Start screening: 📄 [N/Total] Screening: "Title..."
2. Abstract score: Abstract score: X/10
3. Decision: What you're doing next (fetching full text / skipping / etc)

For relevant papers, report findings immediately (adapt to domain):

Medicinal chemistry example:

📄 [15/127] Screening: "Selective BTK inhibitors..."
   Abstract score: 8 → Fetching full text...
   ✓ Found IC50 data for 8 compounds (Table 2)
   ✓ Selectivity data vs 50 kinases (Figure 3)
   → Added to SUMMARY.md

Genomics example:

📄 [23/89] Screening: "Gene expression in liver disease..."
   Abstract score: 9 → Fetching full text...
   ✓ RNA-seq data available (GEO: GSE12345)
   ✓ 123 DEGs identified (Table 1, FDR < 0.05)
   → Added to SUMMARY.md