28sungmin/m4-add-skills/note-accuracy

Note Accuracy

Evaluates the accuracy of unstructured clinical notes and radiology reports by sending each note to Claude CLI for error detection. Covers clinical notes (note_clinical) and radiology reports (note_rad).

When to Use This Skill

• After QUIQ conversion, to assess quality of free-text clinical documentation
• To detect clinically significant errors in admission/discharge notes, surgery notes, radiology impressions, etc.
• As part of LYDUS quality management assessment

SQL Support

Not applicable. Requires Claude CLI for note review.

Filtering Logic

Mapping_info_2 Code Reference

Evaluation Logic

Clinical Notes (note_clinical)

LLM checks 6 error categories per note:

1. Spelling or grammatical error
2. Diagnostic Information Error (counted)
3. Drug Information Error
4. Procedure Information Error (counted)
5. Demographic Information Error
6. Date Information Error

Score = (# of "No" responses among Diagnostic + Procedure) / 2 × 100

• 100% = both categories error-free
• 50% = one of two has an error
• 0% = both have errors

Radiology Reports (note_rad)

LLM identifies one critical error in the Impression section.

Score = 100 if no error, 0 if error found.

Overall Note Accuracy

Unweighted mean across all notes (clinical + radiology combined).

Output

How to Run

import pandas as pd
from scripts.note_accuracy import get_note_accuracy
quiq = pd.read_csv("/path/to/quiq.csv")
df_clinical, df_radiology, result_df, summary_df = get_note_accuracy(
    quiq=quiq
)
mean_accuracy = round(result_df['Accuracy_results'].mean(), 2)
print(f"Note Accuracy (%) = {mean_accuracy}")

As a script with config

# config.yaml
quiq_path: /path/to/quiq.csv
save_path: /path/to/output

python scripts/note_accuracy.py --config config.yaml

Critical Notes

1. Subprocess with timeout — each Claude CLI call runs via subprocess.run with 60-second timeout. Failed rows return an empty result.

1. Clinical score logic — only Diagnostic + Procedure categories count toward the clinical note score (not spelling, drug, demographic, date). This matches the original LYDUS specification.

1. Radiology score parsing — the LLM response is parsed as a Python dict using ast.literal_eval. Responses that cannot be parsed return None (excluded from average).

1. 원본 코드 개선 사항:

• api_call(results=[]) 가변 기본인자 제거 → _call_api_threaded 함수로 리팩터링
• _run_clinical과 _run_radiology에 중복된 api_call/process_rows 내부 함수 → _process_notes 공통 함수로 통합
• get_unstructured_accuracy → get_note_accuracy 로 명칭 변경 (스킬 이름과 일관성)

1. Dependencies — pandas, seaborn, matplotlib, tqdm (LLM: Claude CLI via subprocess)

References

• LYDUS 품질관리 프로그램 활용 가이드라인 (비공개 내부 문서)
• Original Python implementation: LYDUS_Note_Accuracy.py (이성민 작성)