library(mlbench) #this library contains the Pima Indian Diabetes dataset
library(caret)
library(naivebayes)
data(PimaIndiansDiabetes2) #load the dataset
par1 = 9 #column number of target variable
par2 = 'gaussian' #smoothing kernel to be used
par3 = 'nnnnnnnnc' #specify n (numeric) or c (categorical) for each column in the dataset
par4 = 'no' #use repeated cross-validation?
print.naive_bayes <- function (x,...) {
model <- 'Naive Bayes'
n_char <- getOption('width')
str_left_right <- paste0(rep('=', floor((n_char - nchar(model)) / 2)), collapse = '')
str_full <- paste0(str_left_right, ' ', model,' ',
ifelse(n_char %% 2 != 0, '=', ''), str_left_right)
len <- nchar(str_full)
l <- paste0(rep('-', len), collapse = '')
cat('\n')
cat(str_full, '\n', '\n', 'Call:', '\n')
print(x$call)
cat('\n')
cat(l, '\n', '\n')
cat( 'Laplace smoothing:', x$laplace)
cat('\n')
cat('\n')
cat(l, '\n', '\n')
cat(' A priori probabilities:','\n')
print(x$prior)
cat('\n')
cat(l, '\n', '\n')
cat(' Tables:','\n')
tabs <- x$tables
n <- length(x$tables)
indices <- seq_len(min(25,n))
tabs <- tabs[indices]
print(tabs)
if (n > 25) {
cat('\n\n')
cat('# … and', n - 25, ifelse(n - 25 == 1, 'more table\n\n', 'more tables\n\n'))
cat(l)
}
cat('\n\n')
}
x <- na.omit(PimaIndiansDiabetes2) #remove rows with missing data
k <- length(x[1,]) #we could also use ncol(x)
n <- length(x[,1]) #we could also use nrow(x)
myf <- formula(paste(colnames(x)[par1],' ~ .',sep=''))
nb_grid <- expand.grid(usekernel = c(TRUE, FALSE), laplace = c(0, 0.5, 1, 2, 3, 4), adjust = c(0.75, 1, 1.25, 1.5, 1.75, 2, 2.25, 2.5))
fitControl <- trainControl(method = 'repeatedcv', number = 10, repeats = 5)
#please, be patient
if(par4=='no') {
naive_bayes_via_caret <- train(myf, data = x, method = 'naive_bayes', kernel = par2, bw = 'SJ', usepoisson = TRUE, tuneGrid = nb_grid)
}
if(par4=='yes') {
naive_bayes_via_caret <- train(myf, data = x, method = 'naive_bayes', kernel = par2, bw = 'SJ', usepoisson = TRUE, tuneGrid = nb_grid, trControl = fitControl)
}
#show results
naive_bayes_via_caret$results
naive_bayes_via_caret$finalModel$tuneValue
naive_bayes_via_caret$finalModel
z <- cbind(x, predict(naive_bayes_via_caret$finalModel, x, type = 'prob'))
head(z)21 Gaussian Naive Bayes Classifier
21.1 Introduction
The Multinomial Naive Bayes Classifier that was discussed in Chapter 9 made the assumption that the likelihoods are based on Discrete Distributions. If, however, our dataset contains a variable \(X\) with a continuous distribution then it is often convenient to assume it is also normally distributed.
This chapter appears in the Distributions part because the Gaussian Naive Bayes likelihood is a direct application of the Normal density. It complements the model-building chapters by showing how distributional assumptions translate into a concrete classifier.
Suppose there are \(K\) classes that we want to predict1. For each class \(k \in [1, 2, …, K]\) we can compute the mean \(\mu_k\) and variance \(\sigma_k^2\) so that the likelihood for any observed value \(X = \nu\) is
\[ f_{X \mid k}(\nu) = \frac{e^{-\frac{1}{2} \left( \frac{\nu - \mu_k}{\sigma_k} \right)^2} }{\sigma_k \sqrt{2 \pi}} \]
This is a class-conditional density value (not a point probability); for a continuous variable, \(\text{P}(X=\nu \mid k)=0\).
Even if there are multiple variables with a normal distribution it is easy to compute the likelihoods that are needed to apply Bayes’ Theorem and obtain a posterior probability that is used to make a prediction. Of course, it is also possible to combine the likelihoods from the Bernoulli, Multinomial, Poisson, and Normal distribution.
If the normality assumption for a continuous variable is not satisfied, one often uses the empirical Kernel Density instead. Since the Naive Bayes approach can be easily implemented and computed, it is common practice to estimate the classifier for both scenarios (i.e. using Kernel Densities and Normal Distributions). Furthermore, it is also possible to recompute the model for various choices of \(\alpha\) (as was explained in Section 9.4).
21.2 R Module
The Naive Bayes Module can be found on the public website:
The Naive Bayes Module is also available in RFC under the menu “`Models / Manual Model Building”.
If you prefer to compute a Gaussian Naive Bayes model on your local computer, the following code snippet can be used in the R console2:
The R script uses a custom function (print.naive_bayes) to produce the output tables of the model. Furthermore, the data is cleaned through the na.omit function to remove the rows that contain missing data (denoted by NA). Instead of calling the naive_bayes function (from the naivebayes library) directly, the R script uses the train function (from the caret package) to invoke the naive_bayes function for different combinations of kernels, Laplace parameters, and adjustments based on caret’s expand.grid function. Note that the formula function creates the model specification to be estimated (par1 indicates the column that is to be used as target variable).
21.3 Example
We wish to build a model that allows us to predict diabetes, in a population of female patients of Pima Indian heritage (Smith et al. 1988), based on a series of diagnostic measurements.
The R Module uses a random selection of 200 female patients and shows the results of the standard Naive Bayes Classifier (i.e. without optimisation of the Gaussian Kernel Density and Laplace \(\alpha\) values). The output shows the model specification type ~ npreg+glu+bp which means that the variable type is the binary target variable to be predicted (the patient has been diagnosed with diabetes or not) and is explained through three exogenous variables or features (i.e. npreg+glu+bp). The value of \(\alpha\) is zero (Laplace: 0) and the number of rows in our dataset is 200 (Samples: 200). There are three features (Features: 3) which are all treated as Gaussian/Normal distributions. The model uses data-based prior probabilities which are displayed as
- Prior probabilities:
- No: 0.66
- Yes: 0.34implying that 34% of the patients contained in the dataset have the disease.
For each feature the output also shows the descriptive statistics and how they relate to the target variable. For example, the mean of the number of pregnancies npreg for women without diabetes is equal to 2.9167 versus 4.8382 for patients with the disease.
---------------------------------------------------------------------------------
::: npreg (Gaussian)
---------------------------------------------------------------------------------
npreg No Yes
mean 2.916667 4.838235
sd 2.806866 3.972331According to this result, npreg seems to be a good predictor for diabetes because the means are quite different. Near the bottom of the output page the ML Fitted Normal Densities of npreg (for both groups) are shown as well. The predictive power of the feature depends on how much both densities overlap. In other words, the predictive power of the number of pregnancies depends of how far the Likelihoods (\(\text{P}(npreg = \nu | k = \text{no diabetes})\) and \(\text{P}(npreg = \nu | k = \text{diabetes})\)) are apart (which in turn depends on the difference between both mean levels). The question of how far the mean levels should be apart so that the variable has a meaningful contribution in the prediction of diabetes, still remains an open question and will be answered in Hypothesis Testing.
Changing the “Select Variable to Plot” field allows us to examine all the features that have been included in the model. For instance, the variable glu (plasma glucose concentration in an oral glucose tolerance test) promises to be a better predictor than the variable bp (diastolic blood pressure mm Hg).
While it is interesting to examine the individual contribution of individual features, the ultimate purpose of creating a Naive Bayes Classifier is to generate useful predictions. The predictive performance can now be assessed by setting the “Training set %” slider to 90%. This causes the model to be recomputed based on the first 90% of the rows in the dataset (we call this the “training set”). The remaining 10% (i.e. the “test set”) is used to test the quality of the predictions against the actual values. We already know how to evaluate binary classifiers from Chapter 8 which describes the concepts of sensitivity and specificity. After setting the training percentage, the output will show a table with the true positive/negative and false positive/negative values (we call this a “confusion matrix”). In addition, the values for sensitivity and specificity are also displayed (if feasible), along with some other statistics that are discussed in Descriptive Statistics.
The Laplace field allows us to manually select various values for \(\alpha\) and observe how the results of the Naive Bayes Classifier change. In addition, we can also select the density functions with the “Kernel or Poisson” selector (the default is a Gaussian/Normal Density which can be replaced with Kernel functions and/or Poisson Densities).
21.4 Task
Use the Naive Bayes Classifier software shown above and improve its predictive performance. Try to add new features and investigate the effects of changing the Laplace and Kernel settings.
For instance, if we are interested in three classes (i.e. “real”, “fake”, and “mixed” news which contains truths and falsehoods) then we would use \(K = 3\)↩︎
Make sure that the
mlbench,caret, andnaivebayespackages have been installed with theinstall.packagesfunction before running the script.↩︎