skforecast/skforecast-ai/backtesting-configuration

Backtesting Configuration

When to Use

Use this skill to translate a deployment scenario (retraining cadence, forecast horizon, data budget, ingestion delay) into TimeSeriesFold parameters. TimeSeriesFold is the cross-validation strategy passed via the cv argument to backtesting_forecaster (and its multi-series / stats variants) and to the hyperparameter search functions.

from skforecast.model_selection import backtesting_forecaster, TimeSeriesFold
cv = TimeSeriesFold(
    steps=7,                       # forecast horizon
    initial_train_size=365,        # first training window (required)
    refit=False,                   # train once (default)
    fixed_train_size=True,         # rolling window (default)
    gap=0,
)
metric, predictions = backtesting_forecaster(
    forecaster=forecaster,
    y=data['target'],
    cv=cv,
    metric='mean_absolute_error',
)

For fast hyperparameter tuning where multi-step realism is not required, use OneStepAheadFold instead: it validates one step ahead (no recursive prediction), so it is much faster but less representative of multi-step performance. Use TimeSeriesFold for realistic multi-step backtesting.

Related skills

• Prerequisite: forecasting-single-series / forecasting-multiple-series (have a fitted forecaster before backtesting)
• Alongside: metric-selection (choose the metric(s) backtesting reports)
• Alongside: hyperparameter-optimization (the same cv object drives the search functions)
• Next: prediction-intervals (add interval= / interval_method= to backtest uncertainty)

Stop Conditions

Scan before writing code. Each row lists a rule, the symptom when it is broken, and the recovery. Full pitfall catalog: the troubleshooting-common-errors skill.

TimeSeriesFold Parameters

Constraints

• Must produce at least 2 folds: initial_train_size + gap + 2 * steps <= n_observations
• Hard minimum: initial_train_size must be greater than forecaster.window_size, the number of past observations needed to build one training row (max(max_lag, max_size_window_features) plus the differentiation order). Direct forecasters (ForecasterDirect, ForecasterDirectMultiVariate, ForecasterRnn) need at least window_size + steps, using the forecaster's steps. In multi-series forecasters, each series with data in the first window needs more than window_size observations there.
• Forecasters without lags: ForecasterEquivalentDate has window_size = offset * n_offsets (same rule); in ForecasterStats the estimator sets the minimum (window_size is fixed at 1); ForecasterFoundation accepts any value ≥ 1 (context_length is a maximum, not a minimum).
• Recommended: leave a margin above the hard minimum so the model has enough rows to learn from (e.g. two or more seasonal cycles), while keeping at least 2 folds
• gap simulates real-world delay between data availability and forecast usage
• When fixed_train_size=True, the window rolls forward (oldest data discarded). Use for concept drift or when old data is less relevant.
• When fixed_train_size=False, the window expands (all history retained). Use when more data always helps.

Business Scenario Mapping

Retraining frequency

Data freshness vs volume

Deployment gap

Initial training size

Fold stride (test set overlap)

Examples

"I retrain my model every Monday and forecast the next 7 days"

refit = True
fixed_train_size = False  # Keep all history
gap = 0  # No delay
fold_stride = None  # Non-overlapping weeks

"I want to simulate deploying once and seeing how the model degrades"

refit = False  # Train once; fixed_train_size has no effect without refit
gap = 0

"There's a 2-day lag between data ingestion and when forecasts are needed"

gap = 2
refit = True

"I have limited compute — evaluate every other week"

refit = True
skip_folds = 2  # Keep every 2nd fold

"I want maximum evaluation coverage with a 12-step horizon"

# Hard minimum: forecaster.window_size + 1 (window_size + 12 for ForecasterDirect with steps=12)
initial_train_size = forecaster.window_size + margin  # margin: rows to learn from, e.g. 2+ seasonal cycles
refit = False  # Faster; trains once, fixed_train_size has no effect
fold_stride = 1  # Sliding window (many overlapping folds)