158 Introduction to Model Building Strategies
Model building is a sequence of decisions. Each decision — which target to model, which predictors to include, which method to use, how to validate, whether to revise — affects the result and should be documented. This chapter introduces the vocabulary and structure used throughout the rest of this part.
158.1 The Model Building Process
A complete model building workflow consists of stages that are visited in order but can be revisited when diagnostics or domain knowledge require a revision.
| Stage | What happens | Where it is covered |
|---|---|---|
| Data | Choose the dataset, identify the target variable, select candidate predictors, and declare group structure or predictor-availability exceptions when needed | Chapter 63, Chapter 163 |
| Audit | Inspect warnings about missingness, outliers, leakage, delayed variables, and distributional problems before fitting | Chapter 163, Chapter 165 |
| Strategy | Decide on preprocessing, candidate models, validation method, and whether a locked final test is needed | Chapter 163, Chapter 164 |
| Fit | Train one or more candidate models on the training data | Chapter 135, Chapter 136, Chapter 140, Chapter 142 |
| Validate | Evaluate each candidate on held-out data, repeated resamples, and sometimes a separate final confirmation split | Chapter 160, Chapter 164 |
| Diagnose | Inspect residuals, calibration, discrimination, or forecast behavior | Chapter 164 |
| Revise | Test an alternative path and decide whether it should replace the current model | Chapter 164 |
| Report | Export the reasoning, the code, and the results | Chapter 163 |
The point of this table is not that every analysis must pass through every stage. The point is that skipping a stage should be a conscious decision, not an accident.
158.2 Prediction and Explanation as Modeling Goals
The first decision in any modeling workflow is the goal. The choice between prediction and explanation changes how redundancy, validation, interpretability, and diagnostics are weighted.
| Dimension | Prediction | Explanation / Confirmation |
|---|---|---|
| Goal | minimize error on unseen cases | understand which predictors matter and how |
| Preferred models | any model that generalizes well, including ensembles and flexible methods | models with directly interpretable structure (coefficients, tree diagrams, or rule sets) |
| Key metric | held-out RMSE, MAE, AUC, accuracy | interpretability of fitted structure, effect direction, residual structure |
| Variable selection | automated selection and regularization are acceptable | substantive reasoning guides which predictors enter the model |
| Complexity | higher complexity is acceptable if validation supports it | simpler models are preferred when they explain the phenomenon adequately |
This distinction appears in Chapter 136 (where logistic regression serves both goals), in Chapter 60 (where AUC measures predictive discrimination), and in Chapter 140 (where tree structure provides an alternative to coefficient interpretation). The Guided Model Building app (Chapter 163) asks the user to declare this goal before the first model is fitted.
158.3 Single Models and Ensemble Methods
The models discussed in earlier chapters — linear regression (Chapter 135), logistic regression (Chapter 136), conditional inference trees (Chapter 140) — are single models. Each produces one set of predictions from one fitted structure.
An ensemble combines the predictions of multiple models to produce a result that is often more stable or more accurate than any single member. Three families are commonly distinguished:
| Family | Idea | Main effect | Example |
|---|---|---|---|
| Bagging | fit the same model type on resampled copies of the data and average the predictions | reduces variance | conditional random forest (cforest) |
| Boosting | fit models sequentially, each one correcting the errors of the previous one | reduces bias | gradient boosting |
| Stacking | train several different model types and combine their predictions through a second-level model | exploits complementary strengths | stacked generalization |
The Guided Model Building app already includes cforest from the party package as a candidate model. A conditional random forest is a bagging-style ensemble built from many conditional inference trees. Unlike a single tree, the forest is not directly interpretable as a decision diagram, but it typically produces lower prediction error because the averaging across trees reduces the instability that affects any individual tree. The details of this tradeoff are developed in Chapter 142.
Boosting and stacked ensembles are still planned for later in this part.
158.4 Regularization and Hyperparameter Selection
When a model has many predictors relative to the number of observations, ordinary fitting can produce large and unstable coefficients. Regularization addresses this by adding a penalty term to the fitting criterion that shrinks coefficients toward zero.
Two standard forms are widely used:
- Ridge regression adds a penalty proportional to the sum of squared coefficients. It shrinks all coefficients but keeps every predictor in the model.
- Lasso regression adds a penalty proportional to the sum of absolute coefficients. It can shrink some coefficients exactly to zero, effectively performing variable selection as part of the fitting process.
The penalty strength is controlled by a hyperparameter — a model setting that is not estimated from the data itself but must be chosen before fitting begins. The standard way to choose a hyperparameter is by validation: fit the model at several candidate values and select the value that produces the best held-out performance (see Chapter 160).
Hyperparameter selection is not limited to regularization. Any model setting that must be fixed before fitting counts as a hyperparameter: the depth of a tree, the number of trees in a forest, the smoothing bandwidth in a kernel density estimate, or the Laplace correction in a Naive Bayes classifier (Chapter 21). What these settings share is that they control model complexity and should be chosen by validation rather than by convenience.
The next two chapters return to these ideas in concrete form: Chapter 161 treats ridge, lasso, and elastic net, while Chapter 162 treats systematic tuning strategies more generally.
158.5 Manual and Guided Workflows
This part of the book covers two levels of workflow support:
- Manual model building (Chapter 159): the user selects a dataset, fits individual models one at a time, and compares results by reading the output directly. In this handbook, the primary tool for this approach is the app available in the menu
Models / Manual Model Building. - Guided model building (Chapter 163): the Guided Model Building app automates the audit, strategy, fitting, validation, and export stages while keeping every methodological choice visible for review and revision.
Fully automated model building (AutoML) — where the system searches over model families, preprocessing pipelines, and hyperparameters without user intervention — is a third level that exists in practice but is not covered in this handbook. The educational goal here is to make the reasoning behind each modeling decision explicit, which requires the user to remain involved at every stage.
158.6 The Role of Validation
A model that fits the training data well is not necessarily a model that will perform well on new data. The difference between training performance and held-out performance is the central problem that validation addresses. If a model has been allowed to memorize the training data — by using too many parameters, by overfitting noise, or by leaking target information into the predictors — it will appear excellent on training data and fail on new observations.
The next chapter (Chapter 160) introduces the validation methods used throughout this part: single holdout splits, repeated holdout, stratified holdout, rolling-origin validation for time series, and k-fold cross-validation.
In the Guided Model Building app, predictive defaults are therefore not based on one lucky split. They are based on repeated-validation summaries. This matters when you read the stability boxplots in the later guided chapters: the line inside a boxplot is the median resample performance, but the app’s automatic ranking uses the mean repeated-validation performance. If the resample distribution is skewed, a model with the highest median need not be the model with the highest mean.
Validation also has a second role beyond average accuracy: it reveals reliability. Two models can have similar mean RMSE, AUC, or accuracy while one of them varies much more across resamples. A model with slightly weaker average performance but clearly lower variability may be easier to trust in practice because its behavior is less dependent on the particular split.
For binary classification, the same comparison can be read through more than one lens. AUC emphasizes ranking and discrimination across thresholds; AUCPR emphasizes the tradeoff between precision and recall and becomes especially useful when the positive class is rare or when false positives are costly. The guided workflow therefore uses AUC for automatic default ranking but also shows AUCPR as a complementary perspective when precision matters more than sensitivity.
In confirmatory workflows, validation can be made stricter still by reserving a locked final test set. The model is built and revised on the remaining analysis subset, while the final split is kept hidden until the analyst is ready for an explicit last check. This is not needed in every exercise, but it is an important safeguard when the objective is confirmation rather than exploratory optimization.
158.7 Structure of This Part
| Chapter | Focus |
|---|---|
| Introduction to Model Building Strategies (this chapter) | vocabulary, goals, and workflow overview |
| Manual Model Building (Chapter 159) | hands-on classification with the app in the menu Models / Manual Model Building |
| Model Validation (Chapter 160) | holdout, repeated holdout, stratified splits, rolling-origin, comparison metrics |
| Regularization Methods (Chapter 161) | ridge, lasso, elastic net, and coefficient shrinkage by validation |
| Hyperparameter Optimization Strategies (Chapter 162) | grid search, random search, and tuning forests or penalties by held-out performance |
| Guided Model Building in Practice (Chapter 163) | full-screen tabular workflows with audit, strategy, fitting, and export |
| Diagnostics, Revision, and Guided Forecasting (Chapter 164) | revision logic, forecasting workflows, and model promotion |
| Leakage, Target Encoding, and Robust Regression (Chapter 165) | leakage protection, predictor availability, grouped splitting, fold-safe target encoding, and Huber regression |
| Boosting and Stacked Ensemble Methods (planned) | boosting, stacking, and more advanced ensembles beyond cforest |