Skill v1.0.1
currentAutomated scan100/100+2 new
name: skill-literature description: Manage specs/literature/ — scan, convert PDFs/DJVUs, maintain index.json. Invoke for /literature command. allowed-tools: Bash, Read, Write, Edit, AskUserQuestion metadata: version: "1.0.1"
Literature Skill (Direct Execution)
Direct execution skill for managing specs/literature/ directories. Handles PDF/DJVU-to-markdown conversion, index.json maintenance, and filesystem validation. Runs inline using AskUserQuestion for interactivity.
Key behavior: Users see scan results and proposed keywords/summaries BEFORE any files are written. Users confirm chunk boundaries and metadata before conversion completes.
Context References
Reference (do not load eagerly):
- Path:
@specs/literature/index.json- Current literature index - Path:
@specs/702_create_literature_command/reports/01_lit-command.md- Research findings
Execution
Step 1: Parse Arguments
Extract mode, optional file, and optional query from skill args:
# Parse from skill args: "mode={mode} file={file}" or "mode=search query={query text}"mode=$(echo "$ARGUMENTS" | grep -oP 'mode=\K\S+' | head -1)file=$(echo "$ARGUMENTS" | grep -oP 'file=\K\S+' | head -1)# Extract query: everything after "query=" (supports spaces in query text)query=$(echo "$ARGUMENTS" | sed 's/.*query=//' | sed 's/^[[:space:]]*//')# Default to status mode if not specifiedif [ -z "$mode" ]; thenmode="status"fi# Resolve file path (may be relative or absolute)if [ -n "$file" ]; thenif [[ "$file" != /* ]]; thenfile="specs/literature/$file"fifi
Step 2: Generate Session ID
source .claude/scripts/lib/common.shsession_id="$(common_session_id)"# Two-tier fallback: use LITERATURE_DIR if set and exists, otherwise use per-project specs/literature/if [ -n "${LITERATURE_DIR:-}" ] && [ -d "$LITERATURE_DIR" ]; thenlit_dir="$LITERATURE_DIR"elselit_dir="specs/literature"fiindex_file="$lit_dir/index.json"# Determine sources/ prefix for centralized repoif [ -n "${LITERATURE_DIR:-}" ] && [ "$lit_dir" = "$LITERATURE_DIR" ]; thensources_prefix="sources/"elsesources_prefix=""fi
Step 3: Check Tool Availability
Detect available conversion tools:
has_pdftotext=$(which pdftotext 2>/dev/null && echo "yes" || echo "no")has_pdfinfo=$(which pdfinfo 2>/dev/null && echo "yes" || echo "no")has_djvutxt=$(which djvutxt 2>/dev/null && echo "yes" || echo "no")
Step 4: Dispatch to Mode Handler
Route to the appropriate mode:
case "$mode" instatus) handle_status ;;scan) handle_scan ;;convert) handle_convert ;;validate) handle_validate ;;index) handle_index ;;search) handle_search ;;ingest) handle_ingest ;;rebuild) handle_rebuild ;;*)echo "Error: Unknown mode '$mode'. Available: status, scan, convert, validate, index, search, ingest, rebuild"exit 1;;esac
Mode: Ingest
READ .claude/context/project/literature/patterns/literature-ingest-mode.md now and follow it exactly.
Mode: Status (Default)
Show health report: processed vs unprocessed files and index.json state.
Status Step 1: Check Directory
if [ ! -d "$lit_dir" ]; thenecho "## Literature Status"echo ""echo "No specs/literature/ directory found."echo "Create it and add PDF/DJVU files to get started."echo ""echo "**Tool Availability**:"echo "- pdftotext: $has_pdftotext"echo "- djvutxt: $has_djvutxt ($([ "$has_djvutxt" = "no" ] && echo 'install: nix-env -iA nixpkgs.djvulibre' || echo 'available'))"exit 0fi
Status Step 2: Scan for Files
# Find all PDF and DJVU source filespdf_files=$(find "$lit_dir" -name "*.pdf" 2>/dev/null | sort)djvu_files=$(find "$lit_dir" -name "*.djvu" 2>/dev/null | sort)all_source_files="$pdf_files $djvu_files"# Find all markdown files (excluding any in subdirectory source_files/)md_files=$(find "$lit_dir" -name "*.md" -not -path "*/source_files/*" 2>/dev/null | sort)
Status Step 3: Read Index
if [ -f "$index_file" ]; thenentry_count=$(jq '.entries | length' "$index_file" 2>/dev/null || echo "0")indexed_paths=$(jq -r '.entries[].path' "$index_file" 2>/dev/null || echo "")elseentry_count=0indexed_paths=""fi
Status Step 4: Compute Counts
# Count source filespdf_count=$(echo "$pdf_files" | grep -c "\.pdf$" 2>/dev/null || echo 0)djvu_count=$(echo "$djvu_files" | grep -c "\.djvu$" 2>/dev/null || echo 0)md_count=$(echo "$md_files" | grep -c "\.md$" 2>/dev/null || echo 0)# Identify unprocessed source files (PDFs/DJVUs without corresponding .md)unprocessed=()for src in $pdf_files $djvu_files; dobasename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')# Check if any .md file starts with this basenameif ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; thenunprocessed+=("$src")fidoneunprocessed_count=${#unprocessed[@]}processed_count=$(( pdf_count + djvu_count - unprocessed_count ))
Status Step 5: Display Report
## Literature Status**Directory**: specs/literature/**Source Files**: {pdf_count} PDFs, {djvu_count} DJVUs**Converted**: {processed_count} processed, {unprocessed_count} unprocessed**Markdown Files**: {md_count}**Index Entries**: {entry_count}**Tool Availability**:- pdftotext: {has_pdftotext}- djvutxt: {has_djvutxt} {install hint if no}{if unprocessed_count > 0}**Unprocessed Files** ({unprocessed_count}):- {file1}- {file2}...Run `/literature --convert` to convert all, or `/literature --scan` to see details.{end if}{if entry_count > 0 and md_count != entry_count}**Index Health**: {entry_count} indexed entries, {md_count} markdown files — run `/literature --validate` to check consistency.{end if}
Mode: Scan
Find PDF/DJVU files lacking corresponding markdown conversions.
Scan Step 1: Check Directory
Same as Status Step 1 — exit gracefully if directory missing.
Scan Step 2: Find Unprocessed Files
unprocessed=()for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); dobasename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; thenunprocessed+=("$src")fidone
Scan Step 3: Get Page Counts
For each unprocessed file, get page count via pdfinfo:
for src in "${unprocessed[@]}"; doext="${src##*.}"if [ "$ext" = "pdf" ]; thenif [ "$has_pdfinfo" = "yes" ]; thenpages=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}')elsepages="unknown"fielif [ "$ext" = "djvu" ]; thenif [ "$has_djvutxt" = "yes" ]; then# djvused can get page count: djvused -e n file.djvupages=$(djvused -e n "$src" 2>/dev/null || echo "unknown")elsepages="unknown (djvutxt not installed)"fifiecho "- $src ($pages pages)"done
Scan Step 4: Display Results
## Literature Scan Results**Unprocessed Files** ({count}):- {file1} ({N} pages)- {file2} ({N} pages)...**Tool Status**:- pdftotext: {status}- djvutxt: {status} {install hint if unavailable}**Next Steps**:- Convert all: `/literature --convert`- Convert one: `/literature --convert path/to/file.pdf`
If no unprocessed files found:
## Literature Scan ResultsAll source files have been converted. No unprocessed PDFs or DJVUs found.**Files**: {N} PDFs, {M} DJVUs — all converted**Index**: {entry_count} entries in index.jsonRun `/literature --validate` to check index.json consistency.
Mode: Validate
READ .claude/context/project/literature/patterns/literature-validate-mode.md now and follow it exactly.
Mode: Convert
READ .claude/context/project/literature/patterns/literature-convert-mode.md now and follow it exactly.
Mode: Index
READ .claude/context/project/literature/patterns/literature-index-mode.md now and follow it exactly.
Mode: Search
READ .claude/context/project/literature/patterns/literature-search-mode.md now and follow it exactly.
Mode: Import Pipeline (Steps 8-12)
READ .claude/context/project/literature/patterns/literature-import-pipeline-mode.md now and follow it exactly.
Mode: Rebuild
READ .claude/context/project/literature/patterns/literature-rebuild-mode.md now and follow it exactly.
Sub-Index Management
Per-repo sub-index operations for specs/literature-index.json. These operations manage which documents from the global Literature/ repo are relevant to the current project. The sub-index is reference-only: it stores doc_ids and metadata is resolved at runtime from the global index.
All operations assume $LITERATURE_DIR is set (default: ~/Projects/Literature) and the global index exists at $LITERATURE_DIR/index.json.
Init: Create Empty Sub-Index
Creates specs/literature-index.json with empty entries. Safe to run in a project without an existing sub-index.
project_name=$(basename "$(pwd)")today=$(date +%Y-%m-%d)if [ -f "specs/literature-index.json" ]; thenecho "Sub-index already exists at specs/literature-index.json"echo "Current entries: $(jq '.entries | length' specs/literature-index.json) entries"elsejq -n \--arg project "$project_name" \--arg today "$today" \'{"project": $project,"literature_dir": null,"created": $today,"entries": []}' > specs/literature-index.jsonecho "Created specs/literature-index.json for project: $project_name"fi
Add: Append a Document Entry
Validates that doc_id exists in the global index before appending. Idempotent: if the doc_id is already in the sub-index, reports a warning and skips.
# Usage: doc_id="blackburn_2002" relevance="Core reference for modal logic"global_index="${LITERATURE_DIR:-$HOME/Projects/Literature}/index.json"today=$(date +%Y-%m-%d)# Validate doc_id exists in global indexif ! jq -e --arg id "$doc_id" '.entries[] | select(.id == $id)' "$global_index" >/dev/null 2>&1; thenecho "Error: doc_id '$doc_id' not found in global index ($global_index)" >&2exit 1fi# Check if already present in sub-indexif jq -e --arg id "$doc_id" '.entries[] | select(.doc_id == $id)' specs/literature-index.json >/dev/null 2>&1; thenecho "Warning: doc_id '$doc_id' already in sub-index — skipping" >&2exit 0fi# Append entrytmp=$(mktemp)jq --arg id "$doc_id" \--arg rel "${relevance:-}" \--arg today "$today" \'.entries += [{"doc_id": $id,"relevance": (if $rel == "" then null else $rel end),"added": $today,"source": "manual"}]' specs/literature-index.json > "$tmp" && mv "$tmp" specs/literature-index.jsonecho "Added doc_id '$doc_id' to specs/literature-index.json"
Remove: Delete a Document Entry
Removes an entry by doc_id. No-op if doc_id not present.
# Usage: doc_id="blackburn_2002"tmp=$(mktemp)before=$(jq '.entries | length' specs/literature-index.json)jq --arg id "$doc_id" '.entries = [.entries[] | select(.doc_id == $id | not)]' specs/literature-index.json > "$tmp" && mv "$tmp" specs/literature-index.jsonafter=$(jq '.entries | length' specs/literature-index.json)if [ "$before" -eq "$after" ]; thenecho "Warning: doc_id '$doc_id' not found in sub-index — no change"elseecho "Removed doc_id '$doc_id' from specs/literature-index.json"fi
List: Show Entries with Resolved Metadata
Resolves title, authors, and year from the global index for each sub-index entry.
global_index="${LITERATURE_DIR:-$HOME/Projects/Literature}/index.json"entry_count=$(jq '.entries | length' specs/literature-index.json)if [ "$entry_count" -eq 0 ]; thenecho "Sub-index is empty. Run: /literature --subindex add <doc_id>"exit 0fiecho "## Sub-Index Entries ($entry_count)"echo ""while IFS=$'\t' read -r doc_id relevance added source; do# Resolve from global indextitle=$(jq -r --arg id "$doc_id" '.entries[] | select(.id == $id) | .title // "?"' "$global_index" 2>/dev/null | head -1)authors=$(jq -r --arg id "$doc_id" '.entries[] | select(.id == $id) | if (.authors|type)=="array" then (.authors|first) elif (.authors|type)=="string" then .authors else "?" end' "$global_index" 2>/dev/null | head -1)year=$(jq -r --arg id "$doc_id" '.entries[] | select(.id == $id) | (.year // "?") | tostring' "$global_index" 2>/dev/null | head -1)chunk_count=$(jq --arg id "$doc_id" '[.entries[] | select(.parent_doc == $id)] | length' "$global_index" 2>/dev/null || echo 0)status_tag=""if [ "$title" = "?" ] || [ -z "$title" ]; thenstatus_tag=" [NOT IN GLOBAL INDEX]"fiecho "- **$doc_id**${status_tag}"echo " Title: $title ($year) by $authors"[ "$chunk_count" -gt 0 ] && echo " Chunks: $chunk_count"[ -n "$relevance" ] && [ "$relevance" != "null" ] && echo " Relevance: $relevance"echo " Added: $added (source: ${source:-manual})"echo ""done < <(jq -r '.entries[] | [.doc_id, (.relevance // ""), .added, (.source // "manual")] | @tsv' specs/literature-index.json 2>/dev/null)
Validate: Check All doc_ids Exist in Global Index
Migrated: this dangling-ref check is now wired up and reachable as Job 1 (`rebuild_job1_dangling_ref_lint`), specified in .claude/context/project/literature/patterns/literature-rebuild-mode.md (the "Mode: Rebuild" stub above dispatches there), invoked via /literature --rebuild. It previously referenced a --subindex flag that was never parsed anywhere in literature.md (dead documentation). This section is kept only as a pointer so the Sub-Index Management catalogue stays complete; do not add a second, divergent implementation here — edit rebuild_job1_dangling_ref_lint in literature-rebuild-mode.md instead.
Error Handling
See rules/error-handling.md for general patterns. Skill-specific behaviors:
- specs/literature/ missing: Not an error for status/scan — report and suggest next steps
- index.json missing: Initialize with empty structure for convert/index modes; warn for validate
- Conversion engine-tier availability: No longer a Mode: Convert concern — Convert Step 3b
(see .claude/context/project/literature/patterns/literature-convert-mode.md) delegates extraction to literature-convert.sh, which owns the engine-tier ladder (pymupdf4llm -> ... -> pdftotext/djvutxt) and reports total tier exhaustion as its own exit 2 (see the next bullet).
- djvutxt missing: Soft warning — skip DJVU files with message, continue processing PDFs
- Quality gate rejection (`literature-convert.sh` exit 3): Skip the file with the gate's own
reason surfaced via a QUALITY GATE FAILED operator message; listed under Skipped Files, never written to index.json, never chunked.
- Engine-tier exhaustion or hard conversion failure (`literature-convert.sh` exit 1/2): Skip
the file with the engine's own reason from its stderr; listed under Skipped Files, continue processing the remaining targets.
- Invariant: a gate-rejected or hard-failed document is never written to
index.jsonand
never chunked — Mode: Convert (literature-convert-mode.md) only reports it in the Skipped Files summary, it never appears as a partial or silent success.
- jq failure: Use two-step write pattern (write to tmp file, then mv) to avoid corruption
- Git commit failure: Non-blocking — log and continue
- zotero-library.json not found: Exit code 1 from zotero-search.sh — show setup instructions, fall back to index-only search
- zotero-search.sh returns no results: Exit code 2 — continue to index-only search; combine results
- Broken PDF symlink: Validate mode will detect broken symlinks in pdfs/ directory; non-blocking for import
- Duplicate import: Check index for existing bib_key/zotero_key match before importing; show [IMPORTED] tag
Standards Reference
- Token counting:
chars / 4 + 20(matches memory-harvest.sh pattern) - Chunking: content-aware logical splitting at 4,000-line threshold — divide at chapter/section headings; merge small adjacent sections; fall back to mechanical 4,000-line splits when no headings detected
- Source file convention: PDF/DJVU source files are co-located with their converted markdown in the same
specs/literature/directory or subdirectory. Source files are gitignored viaspecs/literature/**/*.pdfandspecs/literature/**/*.djvupatterns. Users must add source files manually after checkout. - Index schema: root
specs/literature/index.jsonusesentries[]with enriched metadata fields (authors, title, year, doc_type, source_format, parent_doc, page_range, bib_key, zotero_key, zotero_path, project_tags) - Drift threshold: >20% change in token count triggers validation warning
- Zotero search: invokes
.claude/extensions/literature/scripts/zotero-search.shwith--format=json --limit=20 {query_terms}; handles exit codes 0 (success), 1 (library not found), 2 (no results) - Import pipeline (
literature-import-pipeline-mode.md): symlink PDF to
$LITERATURE_DIR/pdfs/{citation_key}.pdf, convert via handle_convert() (specified in literature-convert-mode.md) with PREFILL_* env vars, patch index with Zotero fields, git commit (non-blocking)