Chapter 5 Sample for Regression Problems
Here we will perform and solve a classification problem using the mlr3 package.
First, we load the data and create a machine learning task.
library(mlr3)
library(mlr3fselect)
library(mlr3viz)
my_data =read.csv("./winequality-red.csv", sep = ";")
head(my_data)## fixed.acidity volatile.acidity citric.acid residual.sugar chlorides
## 1 7.4 0.70 0.00 1.9 0.076
## 2 7.8 0.88 0.00 2.6 0.098
## 3 7.8 0.76 0.04 2.3 0.092
## 4 11.2 0.28 0.56 1.9 0.075
## 5 7.4 0.70 0.00 1.9 0.076
## 6 7.4 0.66 0.00 1.8 0.075
## free.sulfur.dioxide total.sulfur.dioxide density pH sulphates alcohol
## 1 11 34 0.9978 3.51 0.56 9.4
## 2 25 67 0.9968 3.20 0.68 9.8
## 3 15 54 0.9970 3.26 0.65 9.8
## 4 17 60 0.9980 3.16 0.58 9.8
## 5 11 34 0.9978 3.51 0.56 9.4
## 6 13 40 0.9978 3.51 0.56 9.4
## quality
## 1 5
## 2 5
## 3 5
## 4 6
## 5 5
## 6 5We create a machine learning task:
my_task = as_task_regr(x = my_data, target = "quality")
my_task## <TaskRegr:my_data> (1599 x 12)
## * Target: quality
## * Properties: -
## * Features (11):
## - dbl (11): alcohol, chlorides, citric.acid, density, fixed.acidity,
## free.sulfur.dioxide, pH, residual.sugar, sulphates,
## total.sulfur.dioxide, volatile.acidityNext we choose a learner to perform a regression task:
learner = mlr_learners$get("regr.rpart")
learner## <LearnerRegrRpart:regr.rpart>: Regression Tree
## * Model: -
## * Parameters: xval=0
## * Packages: mlr3, rpart
## * Predict Types: [response]
## * Feature Types: logical, integer, numeric, factor, ordered
## * Properties: importance, missings, selected_features, weightsSince we have 1599 observations, we will split them into test/train using 2:8.
train_set = sample(my_task$nrow, 0.8 * my_task$nrow)
test_set = setdiff(seq_len(my_task$nrow), train_set)We train the model using the test set:
learner$train(my_task, row_ids = train_set)
learner$model## n= 1279
##
## node), split, n, deviance, yval
## * denotes terminal node
##
## 1) root 1279 863.813900 5.630962
## 2) alcohol< 11.55 1081 608.105500 5.488437
## 4) volatile.acidity>=0.385 871 407.628000 5.363949
## 8) sulphates< 0.555 268 102.667900 5.093284
## 16) volatile.acidity>=0.8025 46 33.739130 4.695652 *
## 17) volatile.acidity< 0.8025 222 60.148650 5.175676 *
## 9) sulphates>=0.555 603 276.600300 5.484245
## 18) alcohol< 9.975 294 106.843500 5.292517 *
## 19) alcohol>=9.975 309 148.666700 5.666667
## 38) total.sulfur.dioxide< 14.5 29 21.862070 5.068966
## 76) sulphates< 0.625 15 7.733333 4.533333 *
## 77) sulphates>=0.625 14 5.214286 5.642857 *
## 39) total.sulfur.dioxide>=14.5 280 115.371400 5.728571
## 78) total.sulfur.dioxide>=76 31 7.419355 5.225806 *
## 79) total.sulfur.dioxide< 76 249 99.140560 5.791165 *
## 5) volatile.acidity< 0.385 210 130.995200 6.004762
## 10) sulphates< 0.645 69 25.072460 5.550725 *
## 11) sulphates>=0.645 141 84.737590 6.226950
## 22) alcohol< 9.75 29 14.551720 5.655172 *
## 23) alcohol>=9.75 112 58.250000 6.375000
## 46) pH>=3.255 58 22.568970 6.086207 *
## 47) pH< 3.255 54 25.648150 6.685185 *
## 3) alcohol>=11.55 198 113.863600 6.409091
## 6) sulphates< 0.615 72 34.875000 5.958333 *
## 7) sulphates>=0.615 126 56.000000 6.666667 *We then predict using the test set:
prediction = learner$predict(my_task, row_ids = test_set)
prediction$score()## regr.mse
## 0.4051785We can select the best feature set by using mlr3fselect package and use the auto tuner:
# auto tuner
autos = auto_fselector(
method = "random_search",
learner = lrn("regr.rpart"),
resampling = rsmp("cv"),
measure = msr("regr.mse"),
term_evals = 10,
batch_size = 5
)
autos$train(my_task, row_ids = train_set)## INFO [16:24:49.288] [bbotk] Starting to optimize 11 parameter(s) with '<FSelectorRandomSearch>' and '<TerminatorEvals> [n_evals=10, k=0]'
## INFO [16:24:49.290] [bbotk] Evaluating 5 configuration(s)
## INFO [16:24:49.495] [mlr3] Running benchmark with 50 resampling iterations
## INFO [16:24:49.498] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:49.535] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:49.570] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:49.609] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:49.643] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:49.678] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:49.858] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:49.896] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:49.934] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:49.971] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:50.013] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:50.050] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:50.086] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:50.122] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:50.155] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:50.193] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:50.226] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:50.260] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:50.294] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:50.332] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:50.366] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:50.402] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:50.438] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:50.478] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:50.513] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:50.549] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:50.585] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:50.624] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:50.660] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:50.696] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:50.732] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:50.770] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:50.804] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:50.838] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:50.872] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:50.905] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:50.943] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:50.976] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:51.010] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:51.044] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:51.090] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:51.126] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:51.162] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:51.197] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:51.238] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:51.274] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:51.310] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:51.347] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:51.391] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:51.428] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:51.473] [mlr3] Finished benchmark
## INFO [16:24:51.974] [bbotk] Result of batch 1:
## INFO [16:24:51.975] [bbotk] alcohol chlorides citric.acid density fixed.acidity free.sulfur.dioxide pH
## INFO [16:24:51.975] [bbotk] FALSE FALSE FALSE TRUE FALSE TRUE FALSE
## INFO [16:24:51.975] [bbotk] FALSE FALSE FALSE FALSE FALSE TRUE TRUE
## INFO [16:24:51.975] [bbotk] TRUE TRUE TRUE TRUE FALSE TRUE TRUE
## INFO [16:24:51.975] [bbotk] TRUE FALSE FALSE FALSE FALSE FALSE FALSE
## INFO [16:24:51.975] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE FALSE
## INFO [16:24:51.975] [bbotk] residual.sugar sulphates total.sulfur.dioxide volatile.acidity regr.mse
## INFO [16:24:51.975] [bbotk] TRUE FALSE FALSE FALSE 0.6531754
## INFO [16:24:51.975] [bbotk] FALSE FALSE FALSE FALSE 0.6781009
## INFO [16:24:51.975] [bbotk] TRUE TRUE TRUE TRUE 0.4767217
## INFO [16:24:51.975] [bbotk] FALSE FALSE FALSE TRUE 0.4867314
## INFO [16:24:51.975] [bbotk] TRUE TRUE FALSE TRUE 0.4701894
## INFO [16:24:51.975] [bbotk] runtime_learners uhash
## INFO [16:24:51.975] [bbotk] 0.450 2d902cb9-b8aa-4926-841d-0143082a55b8
## INFO [16:24:51.975] [bbotk] 0.289 50b98f8d-b803-46df-87a1-4b9491af479b
## INFO [16:24:51.975] [bbotk] 0.307 27446aa4-b9f1-40eb-8870-f623f3f4c6f8
## INFO [16:24:51.975] [bbotk] 0.297 67144004-2f85-4dfc-844b-e7cfe81f5e56
## INFO [16:24:51.975] [bbotk] 0.311 9291908c-6160-4c3d-ba82-a67f7d5ce17c
## INFO [16:24:51.976] [bbotk] Evaluating 5 configuration(s)
## INFO [16:24:52.131] [mlr3] Running benchmark with 50 resampling iterations
## INFO [16:24:52.135] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:52.172] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:52.213] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:52.250] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:52.286] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:52.327] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:52.364] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:52.400] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:52.436] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:52.478] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:52.515] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:52.550] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:52.584] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:52.624] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:52.658] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:52.693] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:52.727] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:52.767] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:52.802] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:52.838] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:52.878] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:52.912] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:52.948] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:52.982] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:53.021] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:53.056] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:53.090] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:53.124] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:53.162] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:53.196] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:53.232] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:53.271] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:53.308] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:53.344] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:53.381] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:53.421] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:53.457] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:53.493] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:53.528] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:53.569] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:53.605] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 1/10)
## INFO [16:24:53.641] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 2/10)
## INFO [16:24:53.682] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 3/10)
## INFO [16:24:53.718] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 4/10)
## INFO [16:24:53.753] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 5/10)
## INFO [16:24:53.789] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 6/10)
## INFO [16:24:53.855] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 7/10)
## INFO [16:24:53.891] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 8/10)
## INFO [16:24:53.926] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 9/10)
## INFO [16:24:53.961] [mlr3] Applying learner 'select.regr.rpart' on task 'my_data' (iter 10/10)
## INFO [16:24:54.015] [mlr3] Finished benchmark
## INFO [16:24:54.517] [bbotk] Result of batch 2:
## INFO [16:24:54.519] [bbotk] alcohol chlorides citric.acid density fixed.acidity free.sulfur.dioxide pH
## INFO [16:24:54.519] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE TRUE
## INFO [16:24:54.519] [bbotk] FALSE TRUE TRUE TRUE FALSE TRUE FALSE
## INFO [16:24:54.519] [bbotk] TRUE TRUE FALSE FALSE TRUE FALSE FALSE
## INFO [16:24:54.519] [bbotk] TRUE TRUE TRUE FALSE TRUE TRUE FALSE
## INFO [16:24:54.519] [bbotk] FALSE TRUE TRUE TRUE TRUE TRUE FALSE
## INFO [16:24:54.519] [bbotk] residual.sugar sulphates total.sulfur.dioxide volatile.acidity regr.mse
## INFO [16:24:54.519] [bbotk] TRUE TRUE TRUE TRUE 0.4767217
## INFO [16:24:54.519] [bbotk] FALSE FALSE FALSE TRUE 0.5665966
## INFO [16:24:54.519] [bbotk] FALSE FALSE FALSE FALSE 0.5307512
## INFO [16:24:54.519] [bbotk] TRUE FALSE TRUE FALSE 0.5012905
## INFO [16:24:54.519] [bbotk] FALSE TRUE FALSE TRUE 0.5563537
## INFO [16:24:54.519] [bbotk] runtime_learners uhash
## INFO [16:24:54.519] [bbotk] 0.315 fc43fc7b-0e4e-4a40-94c3-08f4e0f57177
## INFO [16:24:54.519] [bbotk] 0.303 a6d0ff32-3ce5-4c97-9a84-e2de4701a6d2
## INFO [16:24:54.519] [bbotk] 0.295 b38fc6ef-3609-4d0b-826a-67e47bd96393
## INFO [16:24:54.519] [bbotk] 0.311 5c9fa141-daa0-4fab-be64-74ba4e7077ad
## INFO [16:24:54.519] [bbotk] 0.339 dd936f98-bae9-48e4-87bb-99ad7533cd2b
## INFO [16:24:54.521] [bbotk] Finished optimizing after 10 evaluation(s)
## INFO [16:24:54.521] [bbotk] Result:
## INFO [16:24:54.522] [bbotk] alcohol chlorides citric.acid density fixed.acidity free.sulfur.dioxide pH
## INFO [16:24:54.522] [bbotk] TRUE TRUE TRUE TRUE TRUE TRUE FALSE
## INFO [16:24:54.522] [bbotk] residual.sugar sulphates total.sulfur.dioxide volatile.acidity
## INFO [16:24:54.522] [bbotk] TRUE TRUE FALSE TRUE
## INFO [16:24:54.522] [bbotk] features
## INFO [16:24:54.522] [bbotk] alcohol,chlorides,citric.acid,density,fixed.acidity,free.sulfur.dioxide,...
## INFO [16:24:54.522] [bbotk] regr.mse
## INFO [16:24:54.522] [bbotk] 0.4701894## alcohol chlorides citric.acid density fixed.acidity free.sulfur.dioxide
## 1: TRUE TRUE TRUE TRUE TRUE TRUE
## pH residual.sugar sulphates total.sulfur.dioxide volatile.acidity
## 1: FALSE TRUE TRUE FALSE TRUE
## features
## 1: alcohol,chlorides,citric.acid,density,fixed.acidity,free.sulfur.dioxide,...
## regr.mse
## 1: 0.4701894Here we can see that the regr.mse decreases after performing feature selection, that means it does get better after feature selection.
5.1 Visualization
We will use mlr3viz to create some visualizations.
autoplot(my_task)
Autoplot is the default plot for my task. Here it shows a boxplot for the red wine quality. We can see that the interquartile range is between 5-6 and the median is at 5.5. We also view that there are outliers.
The resample result prediction plot and the prediction plot can only compare one or two features:
my_task1 = my_task$select(c("citric.acid","alcohol"))
rr = resample(my_task1, learner, rsmp("cv", folds = 5), store_models = TRUE)## INFO [16:24:54.706] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 1/5)
## INFO [16:24:54.715] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 2/5)
## INFO [16:24:54.725] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 3/5)
## INFO [16:24:54.736] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 4/5)
## INFO [16:24:54.753] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 5/5)autoplot(rr, type = "prediction")
p = plot_learner_prediction(learner, my_task)## INFO [16:24:55.587] [mlr3] Applying learner 'regr.rpart' on task 'my_data' (iter 1/1)print(p)