164  Diagnostics, Revision, and Guided Forecasting

The first fitted model is not the end of the analysis. It is where you start asking whether that model actually holds up. You can test a revised path, compare it with the current one, and then decide whether the revision deserves to replace it.

This chapter shows how to use that loop in practice.

164.1 Open the Revision Sessions Full Screen

WarningFull-screen use

The revision views are especially hard to read in a narrow iframe because they combine comparison tables, plots, and action buttons. Open the forecasting sessions in a new tab.

Open the nottem baseline session Open the nottem revised session Open the AirPassengers revised session

The first link opens a baseline forecasting path. The second opens a prepared revised path. The third remains useful as an additional comparison when you want to study seasonality, Box-Cox transformation, and automatic ARIMA on a more famous monthly series.

The two series used in this chapter represent different forecasting situations:

  • nottem records monthly temperatures at Nottingham Castle and is a compact example for seasonal revision.
  • AirPassengers records monthly international airline passenger totals and is a standard example of level growth together with increasing seasonal amplitude.

164.2 Three Diagnostic Questions

The revision workflow is easier to understand if diagnostics are sorted into three questions:

Diagnostic type Main question Typical app evidence
Data diagnostics was the planned workflow plausible in the first place? missingness, outliers, leakage, ordered rows, high-cardinality warnings
Model diagnostics does the fitted model behave coherently? residual plots, ROC (Chapter 60), confusion matrix (Chapter 59), residual ACF/PACF, forecast intervals
Revision diagnostics does an alternative path materially improve the workflow? before/after comparison tables, model promotion buttons, revised fit summaries

This is the key idea of the chapter: diagnostics are not just something to read. They are what justify moving from one model path to another.

164.3 Predictive Stability Is More Than One Number

For predictive workflows the app now shows a dedicated Predictive stability section. This is important because students often expect one validation number to settle the whole comparison. In practice, the resample distribution itself matters.

The stability view should be read in four layers:

  1. the horizontal boxplots summarize the repeated-validation distribution for every fitted model,
  2. the line inside each box is the median resample performance,
  3. the automatic default uses the mean repeated-validation performance,
  4. the diamond marks the current held-out split shown elsewhere in diagnostics.

These quantities need not agree. If a distribution is skewed, the model with the highest median can differ from the model with the highest mean. A current held-out split can also be kinder or harsher than the repeated average.

It helps to read one row of the stability plot in a fixed order:

  1. look at the horizontal position of the box as a whole,
  2. look at the center line inside the box,
  3. check how wide the box and whiskers are,
  4. then compare the diamond with the rest of that row.

This sequence answers four different questions:

  1. Where does the model usually perform?
  2. What is the median repeated-validation result?
  3. How much does the result vary across resamples?
  4. Is the currently displayed split typical or unusual?

That is why the app pairs the distribution plot with a second plot that compares average performance against variability across resamples. In predictive work, a model with slightly better mean error but much less stable behavior may deserve more caution than a model whose average is only marginally weaker.

For binary classification, the same comparison can also be viewed through AUCPR. The automatic default still uses repeated-validation AUC, but AUCPR is shown as a secondary lens when the analyst cares especially about precision and the positive class is not the majority.

The practical interpretation is:

  • if your main question is pure ranking quality, read the AUC plots first,
  • if false positives are costly and you care more about the precision of the positive predictions, inspect the AUCPR plots as well,
  • if two models are close on average but one is much more stable, that stability difference is itself part of the scientific decision.

164.4 Worked Example: Revising a Forecasting Path with nottem

The nottem series records monthly temperatures at Nottingham Castle. It is a convenient teaching example because the series is short enough to inspect directly, yet structured enough to illustrate forecasting revisions.

Code 
nottem_df <- data.frame(
  month = cycle(nottem),
  temperature = as.numeric(nottem)
)

seasonal_means <- aggregate(temperature ~ month, data = nottem_df, mean)

knitr::kable(
  head(nottem_df, 12),
  caption = "The first 12 observations of the nottem series"
)
The first 12 observations of the nottem series
month temperature
1 40.6
2 40.8
3 44.4
4 46.7
5 54.1
6 58.5
7 57.7
8 56.4
9 54.3
10 50.5
11 42.9
12 39.8
Code 
knitr::kable(
  seasonal_means,
  caption = "Average temperature by month in nottem"
)
Average temperature by month in nottem
month temperature
1 39.695
2 39.190
3 42.195
4 46.290
5 52.560
6 58.040
7 61.900
8 60.520
9 56.480
10 49.495
11 42.580
12 39.530

164.4.1 Baseline Session: What Happens Before Revision?

In the prepared baseline session:

  • row order is already treated as meaningful,
  • the target is temperature,
  • the forecasting goal is Prediction,
  • the starting seasonal period is intentionally left at 1.

That baseline configuration leads the app to a nonseasonal starting path. In the prepared session the selected model is Automatic ARIMA (see Chapter 153) with:

  • mean rolling-origin RMSE: 3.5251
  • mean rolling-origin MAE: 2.9326
  • mean rolling-origin MAPE: 6.0941

Rolling-origin validation (see Section 160.5) is the time-series equivalent of repeated holdout. The app trains the model up to a cutoff point, forecasts the next stretch of observations, then shifts the cutoff forward and repeats. The error metrics above are averages across those origins.

This is a useful baseline because it shows a common mistake: relying on ARIMA to capture patterns that a simpler seasonal model could represent directly.

164.4.2 Revision Logic: What Should Be Tested?

When residual structure remains, the app should trigger a practical question:

Is the current model failing because the workflow still ignores an important structural feature?

In time series, that often means testing:

  • a different seasonal period,
  • an explicit trend + seasonal model,
  • an alternative smoothing path,
  • or automatic ARIMA after transformation.

This is precisely where the earlier time-series chapters become relevant:

164.4.3 Revised Session: What Changes After the Test?

In the prepared revised session, the seasonal period is changed and the selected model becomes Trend + seasonal dummies. Its validation summary improves materially:

  • mean rolling-origin RMSE: 2.4135
  • mean rolling-origin MAE: 1.8353
  • mean rolling-origin MAPE: 4.1093

That is exactly the kind of change that makes a revision worth taking seriously:

  1. the validation error improves,
  2. the structural story becomes clearer,
  3. the revised path is still interpretable.

The app therefore distinguishes between two separate actions:

  • Test revision
  • Make revised model selected

A revision should first survive comparison before it becomes the active path.

164.5 How Promotion Works in the App

After a revision has been tested, the app can:

  • keep the current model,
  • make the revised model selected,
  • or promote a manually compared alternative from the diagnostics screen.

This design prevents the app from silently replacing one workflow with another. The change is explicit, logged, and exportable.

In other words, a revision is a claim that must be backed by evidence before it replaces the current model.

164.6 Locked Final Test for Confirmatory Workflows

In tabular Explanation / Confirmation workflows, the app can reserve a locked final test set before candidate fitting begins. This changes how revision should be interpreted.

The locked split is:

  • excluded from model choice,
  • excluded from revision testing,
  • hidden until the user explicitly reveals the final confirmation check in Export.

This prevents the analyst from repeatedly tuning the workflow against the same final evidence.

The app is also intentionally strict once the locked final test has been revealed. If the user later changes the selected model, retests candidates, or promotes a new revision, the app marks that final check as contaminated. The reason is scientific rather than technical: after the reveal, the split is no longer untouched.

The easiest way to understand this is to think in three phases:

  1. Model-building phase
    • fit candidates
    • inspect diagnostics
    • test revisions
    • do not reveal the locked final test yet
  2. Decision phase
    • choose the model path you are willing to defend
    • stop changing the workflow
  3. Final confirmation phase
    • open Export
    • reveal the locked final test evaluation once
    • interpret it as the last untouched check on the chosen path

If you reveal the locked final test too early and then keep changing the workflow, you have turned the final test into just another tuning device. That is exactly what the lock is meant to prevent.

In the Cars93 confirmatory session from Section 163.4, the workflow is therefore split conceptually into two datasets even though the app shows one session:

  • an analysis subset used for candidate comparison and revision,
  • a locked final subset reserved for the end.

Students should read this as a methodological discipline rather than as a software trick. The goal is to protect the last piece of evidence from being reused until you are genuinely finished with model choice.

164.7 Forecast Uncertainty Is Part of the Story

Forecasting is not only about point predictions. The app displays forecast intervals alongside the central forecast (see Chapter 153 for how ARIMA-based intervals are constructed), so you can distinguish:

  • the central forecast path,
  • the 80% prediction interval,
  • the 95% prediction interval.

This reinforces a habit you should acquire early: a forecast without uncertainty is not a complete forecast.

164.8 When a Revision Should Be Promoted

In practice, a revised path deserves promotion only when at least one of the following holds:

  • it improves validation performance meaningfully,
  • it improves diagnostics meaningfully,
  • it gives a simpler and more coherent scientific explanation,
  • it retains similar performance while becoming easier to interpret.

For predictive models, “similar performance” should be read as more than “the current split looks similar.” The repeated-validation spread matters too. A model with marginally better mean AUC but much wider variability may be less trustworthy operationally than a model with nearly the same mean and much tighter resample behavior. That is exactly the kind of tradeoff that appears when a flexible benchmark such as cforest (Chapter 142) is compared with a simpler model.

These criteria apply beyond forecasting. They also explain why the app may prefer:

  • Huber regression (Section 165.6) over ordinary least squares when outliers dominate the target,
  • a smaller explanatory model over a larger redundant one,
  • or a tree-based explanation when coefficient interpretation becomes unstable.

164.9 Practical Exercises

  1. Open the nottem baseline session and record the selected model and the main rolling-origin error measures.
  2. Open the nottem revised session and explain which revision changed the forecasting path and why the revised model is worth promoting.
  3. Open the AirPassengers revised session and inspect the forecast intervals. How does the uncertainty band change the interpretation of the forecast?
  4. Return to Section 163.5 and inspect the predictive-stability plots. Which model would you choose if you valued average discrimination most, and would your answer change if you valued reliability across resamples more strongly?
  5. Return to Section 163.4 and identify one revision that would deserve testing there. Should it be a transform, a robust model, or a predictor revision? Defend your answer.

Taken together, this chapter and Chapter 163 show how a scientific workflow should behave: first make a transparent initial choice, then test whether that choice survives criticism. The next chapter, Chapter 165, focuses on the workflow protections that keep those choices honest: leakage control, prediction-time availability, grouped splitting, fold-safe target encoding, and Huber robust regression.