28sungmin/m4-add-skills/instance-diversity

Instance Diversity

Calculates instance diversity and Gini-Simpson index per (Original_table_name, Variable_name, Value) combination in a QUIQ-format table. Measures how evenly distributed patients are in terms of how many times they contribute to each clinical value.

When to Use This Skill

• After QUIQ conversion, to assess whether specific clinical values are dominated by a few patients (low diversity) or spread across many patients (high diversity)
• To detect recording bias — e.g., one patient contributing disproportionately many records for a given value
• As part of LYDUS quality management assessment

Filtering Logic

Metrics

For each (Original_table_name, Variable_name, Value) group:

1. Per-patient occurrence count: group by (ClassKey, Patient_id) → Instance_Count per patient
2. Instance Diversity (%): num_patients / total_occurrences × 100

• 높을수록 환자당 평균 발생 횟수가 적음 (여러 환자에 분산)

1. Gini-Simpson Index (%): (1 - Σp²) × 100

• p = patients_with_count_k / total_occurrences
• 환자별 발생 횟수 분포의 Simpson 다양성
• instance_diversity == 1 (모든 환자 1회 발생) → score = 100%

Weighted averages: weighted by total occurrence count across all ClassKey groups.

Output

How to Run

import os
import duckdb
skill_dir = os.path.dirname(os.path.abspath(__file__))
with open(os.path.join(skill_dir, "scripts/duckdb.sql")) as f:
    sql = f.read()
quiq_csv = "/path/to/quiq_3patients.csv"
sql = sql.replace("{quiq_csv}", quiq_csv)
df = duckdb.sql(sql).df()
total_items = df["total_num"].sum()
weighted_instance = round((df["total_num"] * df["Instance Diversity (%)"]).sum() / total_items, 2)
weighted_simpson  = round((df["total_num"] * df["Gini-Simpson Index (%)"]).sum() / total_items, 2)
print(f"Weighted Instance Diversity (%) = {weighted_instance}")
print(f"Weighted Gini-Simpson Index (%) = {weighted_simpson}")
save_path = "/path/to/output"
os.makedirs(save_path, exist_ok=True)
df.to_csv(f"{save_path}/instance_diversity_summary.csv", index=False, encoding="utf-8-sig")
with open(f"{save_path}/instance_diversity_total.txt", "w") as f:
    f.write(f"Weighted Instance Diversity (%) = {weighted_instance}\n")
    f.write(f"Weighted Gini-Simpson Index (%) = {weighted_simpson}\n")
print(f"Saved {len(df):,} rows → {save_path}")

As a script with config

# config.yaml
quiq_path: /path/to/quiq.csv
save_path: /path/to/output

python scripts/instance_diversity.py --config config.yaml

Critical Notes

1. Class Diversity vs Instance Diversity — class diversity는 한 변수 내 값들의 분포를 측정; instance diversity는 특정 값에 기여하는 환자들의 발생 빈도 분포를 측정.

1. Prescription 필터 — Mapping_info_2 LIKE '%drug%' 조건이 추가됨 (처방 정보 중 약물명만 대상).

1. 원본 오타 수정 — 'Weighted Instance Diveristy (%)' → 'Weighted Instance Diversity (%)'

1. None 처리 — 환자가 없거나 총 발생 수가 0인 ClassKey는 None으로 반환됨. 가중 평균 계산 시 0으로 처리.

References

• LYDUS 품질관리 프로그램 활용 가이드라인 (비공개 내부 문서)
• Original Python implementation: LYDUS_Instance_Diversity.py (이성민 작성)
• Simpson, E.H. (1949). Measurement of Diversity. Nature, 163, 688.