mathews-tom/armory/literature-review

Literature Review

Systematic discovery, extraction, and synthesis of academic research on a defined topic.

When to use this skill vs. others

Workflow

Phase 1: Scope Definition

Before searching, establish the review boundaries:

1. Research question — What specific question does the review answer? Vague topics produce vague reviews. "What techniques exist for X" is weaker than "How do methods for X compare on metric Y across domains Z?"
2. Inclusion criteria — Define what counts:

• Date range (e.g., 2020–present)
• Publication type (peer-reviewed, preprints, both)
• Domains/categories (e.g., cs.CL, cs.AI)
• Minimum relevance threshold

1. Exclusion criteria — Define what does not count:

• Tangentially related work
• Non-primary sources (blog posts, tutorials) unless explicitly included
• Duplicate or superseded versions

1. Expected output — What form should the review take? Narrative synthesis, tabular comparison, gap analysis, annotated bibliography, or related-work section?

Present the scope to the user for confirmation before proceeding.

Phase 2: Search & Discovery

Execute searches across available sources. Use multiple queries with varying specificity to avoid single-query blind spots.

Primary source: arXiv (via arxiv-search utility)

uv run --with arxiv python scripts/arxiv_search.py "QUERY" --max-results 30 --sort-by relevance

Vary queries systematically:

• Broad topic query: "retrieval augmented generation"
• Field-scoped query: ti:retrieval AND abs:generation AND cat:cs.CL
• Author-anchored query: au:lewis AND abs:retrieval (when key authors are known)
• Recency query: same terms with --sort-by submitted

Secondary sources (via web search/fetch):

• Semantic Scholar API: https://api.semanticscholar.org/graph/v1/paper/search?query=QUERY&limit=20&fields=title,authors,abstract,year,citationCount,externalIds
• Google Scholar (via web search): site:scholar.google.com QUERY
• Connected Papers (for citation graph exploration): https://www.connectedpapers.com/search?q=QUERY

Snowball strategy:

• Forward snowball: find papers that cite a key paper (Semantic Scholar citations endpoint)
• Backward snowball: follow the references of key papers
• Use citation count as a signal for influence, not quality

Phase 3: Screening & Filtering

For each discovered paper, apply the inclusion/exclusion criteria from Phase 1.

Produce a screening table:

Rules:

• Screen on title + abstract first. Read full paper only for borderline cases.
• When uncertain, include. It is cheaper to drop a paper later than to miss it.
• Track exclusion reasons — they inform the review's limitations section.
• Flag papers that appear in multiple search queries as likely high-relevance.

Phase 4: Data Extraction

For each included paper, extract a structured record:

- id: "2301.07041"
  title: "Paper Title"
  authors: ["Author One", "Author Two"]
  year: 2023
  venue: "NeurIPS 2023"
  research_question: "How does X affect Y?"
  methodology: "Controlled experiment with N=1000"
  key_findings:
    - "Finding 1 with quantitative result"
    - "Finding 2 with effect size"
  limitations: "Single-domain evaluation"
  relevance_to_rq: "Directly compares methods A and B on metric Y"
  citation_count: 142

Extraction discipline:

• Record what the paper demonstrates, not what it claims to demonstrate.
• Distinguish empirical findings (data-backed) from interpretive claims (author's framing).
• Note methodology details that enable cross-paper comparison (datasets, metrics, baselines).
• If the paper is available via pdf_url, read it for extraction. Do not extract from abstracts alone for included papers.

Phase 5: Synthesis

Transform extracted records into structured analysis. The synthesis method depends on the output format requested in Phase 1.

Thematic synthesis — Group papers by theme, approach, or finding:

• Identify recurring themes across papers
• Note where papers agree, disagree, or address different aspects
• Highlight methodological trends (what approaches are gaining/losing traction)

Comparative synthesis — Build comparison tables:

Chronological synthesis — Map the evolution of the field:

• What was the state of knowledge at time T?
• What shifted and why?
• Where is the field heading?

Gap analysis — Identify what is missing:

• Questions raised but not answered by existing work
• Methodological gaps (no one has tried approach X on problem Y)
• Domain gaps (studied in domain A but not B)
• Contradictions between studies that remain unresolved

Phase 6: Output

Produce the review document in the format specified in Phase 1.

Standard structure for a narrative review:

1. Introduction — Research question, scope, and motivation for the review
2. Search methodology — Databases searched, queries used, inclusion/exclusion criteria, screening results (N found → N screened → N included)
3. Findings — Thematic or chronological synthesis of included papers
4. Discussion — Cross-cutting analysis, trends, contradictions, gaps
5. Limitations of this review — Search scope restrictions, potential biases, papers not accessible
6. References — Full citation list for all included papers

Standard structure for a tabular review:

1. Summary table (all included papers with key metadata)
2. Comparison matrix (methods × metrics × results)
3. Gap analysis table (questions × coverage)
4. Reference list

Citation format: Default to Author et al. (Year) in-text with full references at the end. Adapt to the user's specified format (APA, Chicago, IEEE) if requested.

Quality Checks

Before delivering the review, verify:

• [ ] Every included paper has a structured extraction record
• [ ] Every claim in the synthesis is traceable to at least one extracted finding
• [ ] The gap analysis identifies at least one concrete research opportunity
• [ ] The review acknowledges its own limitations (search scope, access, biases)
• [ ] Citation format is consistent throughout
• [ ] No paper is cited that was excluded during screening
• [ ] Contradictions between papers are noted, not silently resolved by picking a side

Edge Cases