leonardodalinky/scider/ml-model-evaluation

Claude Code and agent skill by leonardodalinky/scider

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Model evaluation framework — cross-validation strategies, metric selection by task type, baseline requirements, ablation design, and train/val/test discipline. Use for all ML experiments before reporting results.

Install

npx vskill@latest install leonardodalinky/scider/ml-model-evaluation

SKILL.md

ML Model Evaluation

Overview

Rigorous model evaluation is the foundation of credible ML research. This skill covers the complete evaluation pipeline: how to split data correctly, which metrics to report for which tasks, what baselines are required, how to structure ablation studies, and how to verify that results are statistically meaningful. Apply this skill before writing any results section or claiming model performance.

When to Use This Skill

Use this skill when:

  • Selecting cross-validation strategy for a new experiment
  • Choosing which metrics to compute and report for a task
  • Setting up baseline comparisons before training models
  • Designing ablation studies to identify component contributions
  • Assessing whether train/val/test splits are clean and leak-free
  • Tuning hyperparameters with nested cross-validation
  • Reporting confidence intervals and checking statistical significance
  • Completing the results section of a paper or technical report

Hard Rules (follow for every experiment)

  1. NEVER report accuracy alone for imbalanced datasets — always pair with F1, AUC-PR, or MCC
  2. ALWAYS compare against at least one trivial baseline before claiming model performance
  3. ALWAYS use cross-validation (or a proper held-out test set) — never evaluate on training data
  4. ALWAYS report confidence intervals or standard deviation across folds
  5. For ablation studies: change ONE component at a time and report the delta

Cross-Validation Taxonomy

Choose the CV strategy based on your data structure. Wrong CV choice is one of the most common sources of inflated reported performance.

Standard K-Fold

Use when: data is IID, classes are balanced, no group structure.

python
from sklearn.model_selection import KFold, cross_validate
from sklearn.ensemble import RandomForestClassifier
import numpy as np
kf = KFold(n_splits=5, shuffle=True, random_state=42)
model = RandomForestClassifier(n_estimators=100, random_state=42)
results = cross_validate(
    model, X, y,
    cv=kf,
    scoring=['accuracy', 'f1_macro', 'roc_auc'],
    return_train_score=True
)
print(f"Val accuracy:  {results['test_accuracy'].mean():.3f} ± {results['test_accuracy'].std():.3f}")
print(f"Train accuracy: {results['train_accuracy'].mean():.3f} ± {results['train_accuracy'].std():.3f}")
# Large train-val gap → overfitting
Stratified K-Fold

Use when: classification task with class imbalance. Preserves class distribution in each fold.

python
from sklearn.model_selection import StratifiedKFold
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Pass y to cross_validate so stratification is applied
results = cross_validate(model, X, y, cv=skf, scoring='f1_macro')
Group K-Fold

Use when: samples are not independent — they come from groups (e.g., multiple measurements per patient, sentences from the same document). Leaking groups across folds gives optimistically biased estimates.

python
from sklearn.model_selection import GroupKFold
gkf = GroupKFold(n_splits=5)
# groups: array of group labels, same length as X
results = cross_validate(model, X, y, cv=gkf.split(X, y, groups=groups),
                         scoring='f1_macro')
Time-Series Split

Use when: data has temporal ordering. Future data must never appear in training folds.

Expanding window (default TimeSeriesSplit) — training set grows with each fold:

python
from sklearn.model_selection import TimeSeriesSplit
tscv = TimeSeriesSplit(n_splits=5)
for train_idx, val_idx in tscv.split(X):
    X_train, X_val = X[train_idx], X[val_idx]
    y_train, y_val = y[train_idx], y[val_idx]
    model.fit(X_train, y_train)
    # evaluate on X_val

Sliding window — fixed-size training window (manual):

python
window_size = 1000
step_size = 200
fold_results = []
for start in range(0, len(X) - window_size - step_size, step_size):
    train_idx = slice(start, start + window_size)
    val_idx = slice(start + window_size, start + window_size + step_size)
    model.fit(X[train_idx], y[train_idx])
    score = model.score(X[val_idx], y[val_idx])
    fold_results.append(score)
print(f"Sliding-window val: {np.mean(fold_results):.3f} ± {np.std(fold_results):.3f}")
Nested Cross-Validation (Hyperparameter Tuning + Evaluation)

Use nested CV when you tune hyperparameters AND want an unbiased generalization estimate. The outer loop estimates generalization; the inner loop selects hyperparameters.

python
from sklearn.model_selection import StratifiedKFold, GridSearchCV, cross_val_score
from sklearn.svm import SVC
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)
param_grid = {'C': [0.1, 1, 10], 'kernel': ['rbf', 'linear']}
clf = GridSearchCV(SVC(), param_grid, cv=inner_cv, scoring='f1_macro')
# Outer loop gives unbiased estimate
nested_scores = cross_val_score(clf, X, y, cv=outer_cv, scoring='f1_macro')
print(f"Nested CV F1: {nested_scores.mean():.3f} ± {nested_scores.std():.3f}")

Rule: never use the same CV split for both tuning and evaluation — this is a common source of optimistic bias.

Metric Selection Guide by Task Type

Classification
MetricWhen to UseWhen NOT to Use
AccuracyBalanced classes onlyImbalanced datasets
Balanced AccuracyImbalanced, multi-class—
Precision / RecallWhen false positives or false negatives have asymmetric costsDon't use alone
F1 (macro)Multi-class, treat all classes equallyWhen class sizes matter
F1 (weighted)Multi-class, weight by supportObscures minority class performance
AUC-ROCBinary, probabilistic outputSeverely imbalanced (prefer AUC-PR)
AUC-PRImbalanced binary problemsNot useful for balanced data
MCCBest single metric for imbalanced binaryLess intuitive to report
python
from sklearn.metrics import (
    accuracy_score, balanced_accuracy_score,
    precision_recall_fscore_support, roc_auc_score,
    average_precision_score, matthews_corrcoef,
    confusion_matrix, classification_report
)
import numpy as np
def classification_metrics(y_true, y_pred, y_prob=None, pos_label=1):
    """Compute full classification metric suite."""
    results = {}
    results['accuracy'] = accuracy_score(y_true, y_pred)
    results['balanced_accuracy'] = balanced_accuracy_score(y_true, y_pred)
    p, r, f1, support = precision_recall_fscore_support(
        y_true, y_pred, average=None, zero_division=0
    )
    results['precision_per_class'] = p
    results['recall_per_class'] = r
    results['f1_per_class'] = f1
    for avg in ['macro', 'micro', 'weighted']:
        _, _, f, _ = precision_recall_fscore_support(
            y_true, y_pred, average=avg, zero_division=0
        )
        results[f'f1_{avg}'] = f
    results['mcc'] = matthews_corrcoef(y_true, y_pred)
    results['confusion_matrix'] = confusion_matrix(y_true, y_pred)
    if y_prob is not None:
        if y_prob.ndim == 2:
            # Multi-class: use OvR
            results['auc_roc'] = roc_auc_score(
                y_true, y_prob, multi_class='ovr', average='macro'
            )
            results['auc_pr'] = average_precision_score(
                y_true, y_prob, average='macro'
            )
        else:
            results['auc_roc'] = roc_auc_score(y_true, y_prob)
            results['auc_pr'] = average_precision_score(y_true, y_prob)
    # Imbalance check
    class_counts = np.bincount(y_true)
    imbalance_ratio = class_counts.max() / class_counts.min()
    results['imbalance_ratio'] = imbalance_ratio
    if imbalance_ratio > 3:
        results['imbalance_warning'] = (
            f"Class imbalance detected (ratio {imbalance_ratio:.1f}:1). "
            "Do NOT rely on accuracy. Use F1-macro, AUC-PR, or MCC."
        )
    return results
Regression
MetricFormulaWhen to Use
RMSE√(mean((ŷ-y)²))Default; penalizes large errors; interpretable in target units
MAEmean(\ŷ-y\)When outliers are expected and should not dominate
R²1 - SS_res/SS_totExplained variance; use as secondary metric
MAPEmean(\ŷ-y\/\y\)×100When relative error matters; AVOID when y≈0
Huber LossHybrid MAE/MSERobust to outliers; use δ≈1.35×MAD
python
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import numpy as np
def regression_metrics(y_true, y_pred, huber_delta=1.35):
    """Compute full regression metric suite."""
    results = {}
    results['rmse'] = np.sqrt(mean_squared_error(y_true, y_pred))
    results['mae'] = mean_absolute_error(y_true, y_pred)
    results['r2'] = r2_score(y_true, y_pred)
    # MAPE — guard against zero targets
    nonzero = y_true != 0
    if nonzero.sum() > 0:
        results['mape'] = np.mean(
            np.abs((y_true[nonzero] - y_pred[nonzero]) / y_true[nonzero])
        ) * 100
    else:
        results['mape'] = float('nan')
    # Huber loss
    residuals = y_true - y_pred
    huber = np.where(
        np.abs(residuals) <= huber_delta,
        0.5 * residuals**2,
        huber_delta * (np.abs(residuals) - 0.5 * huber_delta)
    )
    results['huber_loss'] = np.mean(huber)
    return results
Clustering

Distinguish external metrics (require true labels) from internal metrics (only use features):

python
from sklearn.metrics import (
    silhouette_score, davies_bouldin_score,
    adjusted_rand_score, normalized_mutual_info_score
)
def clustering_metrics(X, labels_pred, labels_true=None):
    results = {}
    # Internal metrics (no ground truth needed)
    if len(set(labels_pred)) > 1:  # silhouette requires ≥2 clusters
        results['silhouette'] = silhouette_score(X, labels_pred)
        results['davies_bouldin'] = davies_bouldin_score(X, labels_pred)
        # Lower DB → better; Higher silhouette → better
    # External metrics (require ground truth)
    if labels_true is not None:
        results['ari'] = adjusted_rand_score(labels_true, labels_pred)
        results['nmi'] = normalized_mutual_info_score(
            labels_true, labels_pred, average_method='arithmetic'
        )
        # ARI=1, NMI=1 → perfect; ARI=0 → random
    return results
Ranking / Retrieval
python
import numpy as np
def ndcg_at_k(relevance_scores, k):
    """relevance_scores: list of relevance for ranked items (desc order)."""
    def dcg(scores):
        return sum(rel / np.log2(i + 2) for i, rel in enumerate(scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)
    return dcg(relevance_scores) / (dcg(ideal) + 1e-10)
def mean_average_precision(relevant_sets, ranked_lists):
    """AP averaged over queries."""
    aps = []
    for relevant, ranked in zip(relevant_sets, ranked_lists):
        hits, ap = 0, 0.0
        for i, item in enumerate(ranked, 1):
            if item in relevant:
                hits += 1
                ap += hits / i
        aps.append(ap / max(len(relevant), 1))
    return np.mean(aps)
def mean_reciprocal_rank(relevant_sets, ranked_lists):
    rrs = []
    for relevant, ranked in zip(relevant_sets, ranked_lists):
        for i, item in enumerate(ranked, 1):
            if item in relevant:
                rrs.append(1 / i)
                break
        else:
            rrs.append(0.0)
    return np.mean(rrs)
Generation / Seq2Seq
python
# Install: pip install evaluate bert_score
import evaluate
# BLEU
bleu = evaluate.load("bleu")
result = bleu.compute(predictions=["the cat sat"], references=[["the cat sat on the mat"]])
print(f"BLEU: {result['bleu']:.4f}")
# ROUGE
rouge = evaluate.load("rouge")
result = rouge.compute(predictions=["the cat sat"], references=["the cat sat on the mat"])
# rouge1, rouge2, rougeL keys

This is the start of the file. Read the full SKILL.md on GitHub.

Security

  • Security scanPassed, 100 of 100Pattern checks on the skill's files.
  • Model reviewPassed

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

Repository stars
88
Repository forks
7
7-day trend score
22

Stars and forks are for the whole GitHub repository, not this skill alone. Refreshed 28 Oct 2026, 11 days ago.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/leonardodalinky/scider/ml-model-evaluation/badge)](https://verified-skill.com/skills/leonardodalinky/scider/ml-model-evaluation)

Something wrong with this skill or its page? Tell us.