How do I split time series data into training and test sets?

Why random splits mislead

A standard random split (shuffle) puts neighbouring hours into both the training and the test sets. One hour’s value is almost the same as the next hour’s. The model learns that neighbourhood, so the test score looks better than the model will perform in service. In production the model must predict forward, and no such neighbour exists.

A correct split respects order: the training period ends before the test period begins. A single cut point is the simplest version. One test period, however, may reflect one event’s luck or one unusual season, so several consecutive splits give a more reliable picture.

Hypothetical year of hourly data

Suppose a pump has 365 days of hourly records, which is 365 × 24 = 8 760 rows. The first 300 days (7 200 rows) are used for training and the last 65 days (1 560 rows) for testing. 7 200 + 1 560 = 8 760, so no rows are lost.

Walk-forward evaluation moves the cut point forward:

• Fold 1: train on days 1–200, test on days 201–230.
• Fold 2: train on days 1–230, test on days 231–260.
• Fold 3: train on days 1–260, test on days 261–290.

Each test window is 30 days. The three folds give 90 days of out-of-sample evaluation, and the spread of scores across folds is more informative than any single number.

Leakage and common mistakes

Leakage happens when information from the test period enters training. Three routes are common:

• A centred window: a 25-hour centred rolling mean at time t includes the 12 hours after t, which the model could not know in service. A trailing window uses only the past.
• A scaler fitted on all data: the mean and standard deviation then include test-period values.
• Overlapping windows: a window that spans the boundary places the same event on both sides of the split.

The fix is to fit every transformation inside each fold, using only that fold’s training data, and to check rolling features for any reference to future values.

Follow-up questions