28sungmin/m4-add-skills/note-fidelity

Note Fidelity

Evaluates the completeness (fidelity) of unstructured clinical notes and radiology reports by asking an LLM whether each required template item is present in the note. Covers clinical notes (note_clinical) and radiology reports (note_rad).

When to Use This Skill

• After QUIQ conversion, to assess whether clinical notes contain all expected structured items
• To detect notes that are missing key sections (e.g., no "Diagnosis" in an admission note)
• As part of LYDUS quality management assessment

SQL Support

Not applicable. Requires Claude CLI for note content analysis.

Filtering Logic

Notes with unrecognized Mapping_info_2 codes are skipped ("Invalid mapping or template missing").

Report Templates

Clinical Notes

Radiology Reports

Scoring Logic

LLM response classifies each template item as mentioned or not mentioned.

Fidelity Score (%) = (1 - not_mentioned / total_items) × 100

• Clinical: items extracted via regex :\s*("[^"]+"|[^:\n]+)
• Radiology: items extracted by splitting on newlines

Overall Note Fidelity = unweighted mean across all scored notes.

Output

How to Run

import pandas as pd
from scripts.note_fidelity import get_note_fidelity
quiq = pd.read_csv("/path/to/quiq.csv")
df_clinical, df_radiology, result_df, summary_df, \
mean_clinical, std_clinical, mean_radiology, std_radiology = get_note_fidelity(
    quiq=quiq
)
mean_fidelity = round(result_df['Fidelity_results'].mean(), 2)
print(f"Note Fidelity (%) = {mean_fidelity}")
print(f"Clinical: {mean_clinical} ± {std_clinical}")
print(f"Radiology: {mean_radiology} ± {std_radiology}")

As a script with config

# config.yaml
quiq_path: /path/to/quiq.csv
save_path: /path/to/output

python scripts/note_fidelity.py --config config.yaml

Critical Notes

1. Retry logic — each LLM call retries up to 3 times on failure. Rows still failing after retries are recorded as "Failed after retries" and excluded from scoring.

1. Invalid mapping — notes with Mapping_info_2 codes not in the template list return "Invalid mapping or template missing" and are excluded from scoring.

1. Note Fidelity vs Note Accuracy — Fidelity measures what items are present (template completeness); Accuracy measures whether the content is correct (error detection).

1. 원본 코드 개선 사항:

• process_row + retry_process_row inner 함수가 _run_clinical / _run_radiology 양쪽에 중복 → _call_with_retry 공통 함수로 통합
• 오타 수정: Present ilness → Present illness, Familty history → Family history, Physical exammination → Physical examination
• get_unstructured_fidelity → get_note_fidelity (스킬 이름과 일관성)
• 템플릿/매핑 딕셔너리를 모듈 수준 상수로 분리

1. Dependencies — pandas, seaborn, matplotlib, tqdm (LLM: Claude CLI via subprocess)

References

• LYDUS 품질관리 프로그램 활용 가이드라인 (비공개 내부 문서)
• Original Python implementation: LYDUS_Note_Fidelity.py (이성민 작성)