andyzhuang/openlife/semantic-sim

🦖 Semantic Similarity Index

Measure how isolated or connected disease research is across the global biomedical literature, using PubMedBERT embeddings on PubMed abstracts spanning 175 GBD diseases.

What it does

1. Takes a disease list (GBD taxonomy) as input
2. Retrieves PubMed abstracts (2000-2025) for each disease with quality filtering
3. Generates 768-dimensional PubMedBERT embeddings for every abstract
4. Computes four semantic equity metrics per disease:

• Semantic Isolation Index (SII): average cosine distance to k-nearest disease neighbours; higher = more isolated, less connected research
• Knowledge Transfer Potential (KTP): cross-disease centroid similarity; higher = more potential for research spillover
• Research Clustering Coefficient (RCC): within-disease embedding variance; higher = more diverse research approaches
• Temporal Semantic Drift: cosine distance between yearly centroids; measures how research focus evolves

1. Generates publication-quality multi-panel figures:

• Panel A: Semantic isolation by disease category (boxplot)
• Panel B: Top 20 most semantically isolated diseases (bar chart, NTD/Global South colour-coded)
• Panel C: Semantic isolation vs research volume (scatter with regression)
• Panel D: NTD vs non-NTD significance test (Welch's t-test, Cohen's d)

1. Produces a markdown report with all metrics, rankings, and reproducibility bundle

Why this exists

If you ask ChatGPT to "measure research neglect for diseases," it will:

• Not know which embedding model to use for biomedical text
• Hallucinate metrics that sound plausible but have no methodological grounding
• Skip quality filtering (year coverage, abstract coverage, minimum papers)
• Not handle MPS acceleration or checkpointed batch processing
• Produce a single scatter plot with no disease classification

This skill encodes the correct methodological decisions:

• Uses PubMedBERT (the gold-standard biomedical language model)
• Fetches from PubMed with exponential backoff and NCBI rate limiting
• Quality filters: year coverage >= 70%, abstract coverage >= 95%, minimum 50 papers
• Batch embedding with Apple MPS acceleration and CPU fallback
• Checkpointed processing (resume after interruption)
• HDF5 storage with gzip compression and SHA-256 checksums
• Classification against WHO NTD list and Global South priority diseases
• Statistical significance testing (Welch's t-test, Cohen's d)

Key Finding

Neglected tropical diseases (NTDs) are significantly more semantically isolated than other conditions (P < 0.001, Cohen's d = 0.8+). They exist in knowledge silos with limited cross-disciplinary research bridges. The 25 most isolated diseases are disproportionately Global South priority conditions.

Pipeline

05-00-heim-sem-setup.py     # Validate environment, create directories
05-01-heim-sem-fetch.py     # Retrieve PubMed abstracts (checkpointed)
05-02-heim-sem-embed.py     # Generate PubMedBERT embeddings (MPS/CPU)
05-03-heim-sem-compute.py   # Compute SII, KTP, RCC, temporal drift
05-04-heim-sem-figures.py   # Generate publication figures
05-05-heim-sem-integrate.py # Merge with biobank + clinical trial dimensions

Demo (works out of the box)

python semantic_sim.py --demo --output demo_report

The demo uses pre-computed embeddings and metrics for 175 GBD diseases and generates the full 4-panel figure instantly.

Example Output

Semantic Similarity Index
=========================
Diseases analysed: 175
Total PubMed abstracts: 13,100,000
Embedding model: PubMedBERT (768-dim)
 
Metric Ranges:
  SII: 0.0412 - 0.1893
  KTP: 0.6234 - 0.9187
  RCC: 0.0891 - 0.3421
 
Key Finding:
  NTDs show +38% higher semantic isolation
  P < 0.0001, Cohen's d = 0.84
  14/25 most isolated diseases are Global South priority
 
Figures saved to: demo_report/
  Fig5_Semantic_Structure.png (300 dpi)
  Fig5_Semantic_Structure.pdf (vector)
 
Reproducibility:
  commands.sh | environment.yml | checksums.sha256

Interpretation Guide

• High SII: Disease research exists in a knowledge silo; limited cross-disciplinary bridges
• Low KTP: Research on this disease has few methodological overlaps with others
• High RCC: Diverse research approaches within the disease (many subtopics)
• High Temporal Drift: Research focus has shifted significantly over time
• NTDs shown in red, Global South diseases in orange, others in grey
• The scatter plot (Panel C) reveals the inverse relationship between research volume and isolation

Citation

If you use this skill in a publication, please cite:

• Corpas, M. et al. (2026). HEIM: Health Equity Index for Measuring structural bias in biomedical research. Under review.
• Corpas, M. (2026). OpenLife. https://github.com/OpenLife/OpenLife