skforecast/skforecast-ai/metric-selection

Name: metric-selection
Author: skforecast

Trust tier 2

community

LLM review pending

By skforecast
AI/ML
skforecast/skforecast-ai
skforecast_ai/skills/metric-selection/SKILL.md
Updated 23 Sep 2026
Version 1.0.01 version recorded

Source check pending

Checking that the SKILL.md file is still in the repository.

Guides selection of the most appropriate evaluation metric(s) for a forecasting task based on forecaster type, prediction output type, data characteristics, and multi-series aggregation needs. Use when the user asks which metric to use, how to evaluate forecast quality, or needs help configuring the `metric` parameter in backtesting or hyperparameter search.

Install

npx vskill@latest install skforecast/skforecast-ai/metric-selection

SKILL.md

The same SKILL.md was published earlier as skforecast/skforecast/metric-selection.

Metric Selection

When to Use

Use this skill when the user needs help choosing an evaluation metric, comparing metrics, understanding trade-offs between error measures, or configuring the metric parameter in backtesting_forecaster, bayesian_search_forecaster, or any other model selection function.

Related skills

Prerequisite: choosing-a-forecaster (the forecaster type determines which metrics apply)
Alongside: baseline-forecasting (MASE < 1 means the model beats the naive baseline)
Next: hyperparameter-optimization (the chosen metric drives the search objective)
Next: prediction-intervals (probabilistic metrics evaluate interval quality)

Quick Recommendations

If unsure, start here:

Point forecast metrics (pass as metric= in backtesting/search):

Task	Default metric	Why
General purpose	`'mean_absolute_error'`	Interpretable, robust to outliers, works at any scale
Compare across series	`'mean_absolute_scaled_error'`	Scale-independent — the only fair comparison across differently-scaled series
Classification	`'balanced_accuracy_score'`	Handles class imbalance common in time series classification

Probabilistic metrics (computed post-hoc on interval/quantile predictions):

Task	Metric	Why
Interval calibration	`calculate_coverage`	Check if actual coverage matches the nominal level
Interval quality (overall)	`crps_from_predictions`	Evaluates sharpness and calibration together
Quantile quality (foundation models)	`crps_from_quantiles`	Proper scoring rule for quantile predictions
Single-interval quality	`winkler_score`	Proper interval score; tune the miscoverage penalty via `alpha`
Full-distribution quality	`weighted_interval_score`	Scores K intervals + median in one number; approximates CRPS
Single-quantile optimization	`create_mean_pinball_loss(alpha)`	Can be passed as `metric=` to optimize for a specific quantile

Step 1 — What Are You Evaluating?

What is your forecaster producing?
│
├─► Point forecasts (single value per step)
│   └─► Go to Step 2
│
├─► Prediction intervals (lower_bound, upper_bound)
│   └─► Go to Step 3
│
├─► Quantile predictions (multiple quantile levels)
│   └─► Go to Step 3
│
├─► Class labels (ForecasterRecursiveClassifier)
│   └─► Go to Step 4
│
└─► Multi-series with aggregation needs
    └─► Go to Step 5 (then return to Step 2 or 3 for the base metric)

Step 2 — Point Forecast Metrics

Decision Table

Criterion	Recommended metric	Avoid
General purpose	MAE (`'mean_absolute_error'`)	—
Penalize large errors more	MSE (`'mean_squared_error'`)	—
Robust to outliers	MAE or MedAE (`'median_absolute_error'`)	MSE (inflated by outliers)
Need percentage interpretation	SMAPE (`'symmetric_mean_absolute_percentage_error'`)	MAPE if data has zeros
Data contains zeros or near-zero values	MAE, MASE, RMSSE	MAPE (divides by y_true → infinite)
Compare across different-scale series	MASE (`'mean_absolute_scaled_error'`) or RMSSE (`'root_mean_squared_scaled_error'`)	MAE, MSE (scale-dependent)
Target is always positive	MSLE (`'mean_squared_log_error'`)	— (use when relative errors matter more than absolute)
Interpretable baseline comparison	MASE (value < 1 means better than naive forecast)	—

Metric Properties

Full property matrix (scale independence, robustness, zero handling, y_train requirement, range and interpretation) for every metric: references/metric-compatibility.md.

Using Metrics in Code

Direct computation (train/test split)

For a simple train/test evaluation, import the metric and call it directly. Sklearn metrics are imported from sklearn.metrics; skforecast-specific metrics (mean_absolute_scaled_error, root_mean_squared_scaled_error, symmetric_mean_absolute_percentage_error) from skforecast.metrics.

python

from sklearn.metrics import mean_squared_error
from skforecast.metrics import mean_absolute_scaled_error
forecaster.fit(y=data_train)
predictions = forecaster.predict(steps=36)
# Sklearn metrics: func(y_true, y_pred)
error_mse = mean_squared_error(y_true=data_test, y_pred=predictions)
# Skforecast metrics that need y_train: func(y_true, y_pred, y_train)
error_mase = mean_absolute_scaled_error(
    y_true=data_test, y_pred=predictions, y_train=data_train
)

Inside backtesting

When using backtesting or hyperparameter search, pass metrics as strings — no import needed, and y_train is handled automatically:

python

from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold
cv = TimeSeriesFold(steps=12, initial_train_size=len(y_train), refit=False)
# Single metric (pass as string)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric='mean_absolute_error',
)
# Multiple metrics at once
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric=['mean_absolute_error', 'mean_absolute_scaled_error'],
)
# metric is a DataFrame with one column per metric

MASE and RMSSE require training data to compute the naive forecast baseline. When passed as a string or callable, skforecast handles y_train automatically — it detects whether the function signature includes a y_train parameter and provides the training data if so.

python

# Both options work identically in backtesting — y_train is handled internally
metric = 'mean_absolute_scaled_error'            # Option 1: as string
metric = mean_absolute_scaled_error              # Option 2: as callable (import from skforecast.metrics)

Step 3 — Probabilistic Forecast Metrics

Use these metrics when evaluating prediction intervals or quantile forecasts.

Metric	Evaluates	Input	Use case
`calculate_coverage`	Calibration	y_true, lower_bound, upper_bound	Check if actual coverage matches nominal level
`crps_from_predictions`	Calibration + sharpness	y_true (scalar), y_pred (array of bootstrap samples)	Evaluate bootstrapped interval quality
`crps_from_quantiles`	Calibration + sharpness	y_true (scalar), pred_quantiles, quantile_levels	Evaluate quantile predictions (foundation models)
`winkler_score`	Calibration + sharpness (single interval)	y_true, lower_bound, upper_bound, alpha	Evaluate one interval; penalty tuned by `alpha`
`weighted_interval_score`	Calibration + sharpness (full distribution)	y_true, y_pred, lower_bounds, upper_bounds, alphas	Evaluate K intervals + median in one CRPS-like score
`create_mean_pinball_loss(alpha)`	Single-quantile accuracy	y_true, y_pred (at quantile alpha)	Evaluate a specific quantile forecast

Coverage

Coverage measures the proportion of true values that fall within the predicted interval. Target: match the nominal level (e.g., 90% interval should have ~90% coverage).

python

from skforecast.metrics import calculate_coverage
coverage = calculate_coverage(
    y_true=y_test,
    lower_bound=predictions['lower_bound'],
    upper_bound=predictions['upper_bound'],
)
# Ideal: coverage ≈ 0.90 for a 90% interval
# coverage >> 0.90 → intervals too wide (not sharp)
# coverage << 0.90 → intervals too narrow (miscalibrated)

CRPS (Continuous Ranked Probability Score)

CRPS is a proper scoring rule that rewards both calibration and sharpness. Lower is better.

python

from skforecast.metrics import crps_from_predictions, crps_from_quantiles
import numpy as np
# From bootstrap samples (e.g., predict_bootstrapping output)
crps = crps_from_predictions(
    y_true=100.0,
    y_pred=np.array([98.2, 101.5, 99.8, 102.1, 97.5])  # Bootstrap predictions
)
# From quantile predictions (e.g., foundation model output)
crps = crps_from_quantiles(
    y_true=100.0,
    pred_quantiles=np.array([90.0, 95.0, 100.5, 105.0, 110.0]),
    quantile_levels=np.array([0.1, 0.25, 0.5, 0.75, 0.9]),
)

Winkler Score (Interval Score)

Scores a single prediction interval: it rewards narrow intervals but adds a penalty, scaled by alpha, whenever the true value falls outside. Strictly proper scoring rule; lower is better. alpha must match the interval's nominal level (e.g. an 80% interval given by the 10th/90th percentiles uses alpha=0.2).

python

from skforecast.metrics import winkler_score
# 80% interval -> alpha = 0.2
score = winkler_score(
    y_true=y_test,
    lower_bound=predictions['lower_bound'],
    upper_bound=predictions['upper_bound'],
    alpha=0.2,
)

Weighted Interval Score (WIS)

Generalises the Winkler Score to K intervals plus the median forecast, producing a single number that behaves like a discrete approximation of CRPS. Ideal when a model outputs several quantile levels (foundation models, ForecasterStats, or bootstrapping). y_pred is the median (0.5 quantile); alphas lists the K interval levels and must line up, by position, with the columns of the bound arrays. Lower is better.

python

import numpy as np
from skforecast.metrics import weighted_interval_score
# Two intervals: 80% (alpha=0.2) and 95% (alpha=0.05)
score = weighted_interval_score(
    y_true=y_test.to_numpy(),
    y_pred=predictions_median,                   # 0.5 quantile
    lower_bounds=np.column_stack([q10, q025]),   # columns match `alphas` order
    upper_bounds=np.column_stack([q90, q975]),
    alphas=[0.2, 0.05],
)

Note: Like calculate_coverage and CRPS, winkler_score and
weighted_interval_score are computed post-hoc. Their signatures differ from
func(y_true, y_pred), so they cannot be passed as a metric= string in
backtesting or hyperparameter search.

Pinball Loss (Quantile Loss)

Evaluates a single quantile forecast. Use create_mean_pinball_loss(alpha) to create a metric function for a specific quantile level.

python

from skforecast.metrics import create_mean_pinball_loss
# Create metric for the 90th percentile
pinball_90 = create_mean_pinball_loss(alpha=0.9)
# Use in backtesting (as callable)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric=pinball_90,
)

Step 4 — Classification Metrics

For ForecasterRecursiveClassifier only. These metrics evaluate predicted class labels, not continuous values.

Metric	Best for
`'accuracy_score'`	Balanced classes (equal importance per class)
`'balanced_accuracy_score'`	Imbalanced classes (common in time series — e.g., rare events)
`'f1_score'`	When both precision and recall matter
`'precision_score'`	When false positives are costly
`'recall_score'`	When false negatives are costly (e.g., missing a spike)

python

from skforecast.recursive import ForecasterRecursiveClassifier
from sklearn.ensemble import RandomForestClassifier
from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold
forecaster = ForecasterRecursiveClassifier(
    estimator=RandomForestClassifier(n_estimators=100, random_state=123),
    lags=24,
)
# y contains class labels (e.g., 'low', 'medium', 'high'); encoding is handled internally
cv = TimeSeriesFold(steps=10, initial_train_size=len(y) - 100, refit=False)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=y,
    cv=cv,
    metric='balanced_accuracy_score',
)

Step 5 — Multi-Series Metric Aggregation

When predicting multiple series, skforecast computes per-series metrics and can aggregate them into a single score. The aggregation options are:

This is the start of the file. Read the full SKILL.md on GitHub.

Security

Security scanPassed, 100 of 100Pattern checks on the skill's files.
Model reviewNot runA model review is scheduled for the next rescan.

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingskforecast_ai/skills/metric-selection/SKILL.md

Popularity

Repository stars: 50
Repository forks: 6
7-day trend score: 20

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/skforecast/skforecast-ai/metric-selection/badge)](https://verified-skill.com/skills/skforecast/skforecast-ai/metric-selection)

Something wrong with this skill or its page? Tell us.