# 13 HPT: sklearn RandomForestClassifier VBDP Data

This document refers to the following software versions:

- `python`: 3.10.10

    pip list | grep  "spot[RiverPython]"

    spotPython                 0.3.0
    spotRiver                  0.0.93

    Note: you may need to restart the kernel to use updated packages.

`spotPython` can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.

    !pip install spotPython

- Uncomment the following lines if you want to for (re-)installation the latest version of `spotPython` from gitHub.

    # import sys
    # !{sys.executable} -m pip install --upgrade build
    # !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

## 13.1 Step 1: Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time and the initial design size.

    MAX_TIME = 1
    INIT_SIZE = 5
    ORIGINAL = False

    import os
    import copy
    import socket
    from datetime import datetime
    from dateutil.tz import tzlocal
    start_time = datetime.now(tzlocal())
    HOSTNAME = socket.gethostname().split(".")[0]
    experiment_name = '16-rf-sklearn' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
    experiment_name = experiment_name.replace(':', '-')
    print(experiment_name)
    if not os.path.exists('./figures'):
        os.makedirs('./figures')

    16-rf-sklearn_p040025_1min_5init_2023-07-04_01-48-35

    import warnings
    warnings.filterwarnings("ignore")

## 13.2 Step 2: Initialization of the Empty `fun_control` Dictionary

Caution: Tensorboard does not work under Windows

- Since tensorboard does not work under Windows, we recommend setting the parameter `tensorboard_path` to `None` if you are working under Windows.

    from spotPython.utils.init import fun_control_init
    fun_control = fun_control_init(task="classification",
        tensorboard_path="runs/16_spot_hpt_sklearn_classification")

## 13.3 Step 3: PyTorch Data Loading

### 13.3.1 Load Data: Classification VBDP

    import pandas as pd
    if ORIGINAL == True:
        train_df = pd.read_csv('./data/VBDP/trainn.csv')
        test_df = pd.read_csv('./data/VBDP/testt.csv')
    else:
        train_df = pd.read_csv('./data/VBDP/train.csv')
        # remove the id column
        train_df = train_df.drop(columns=['id'])

    from sklearn.preprocessing import OrdinalEncoder
    n_samples = train_df.shape[0]
    n_features = train_df.shape[1] - 1
    target_column = "prognosis"
    # Encoder our prognosis labels as integers for easier decoding later
    enc = OrdinalEncoder()
    train_df[target_column] = enc.fit_transform(train_df[[target_column]])
    train_df.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
    print(train_df.shape)
    train_df.head()

    (707, 65)

| x1 | x2 | x3 | x4 | x5 | x6 | x7 | x8 | x9 | x10 | ... | x56 | x57 | x58 | x59 | x60 | x61 | x62 | x63 | x64 | prognosis
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 3.0
1 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 7.0
2 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 3.0
3 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 10.0
4 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 6.0

5 rows × 65 columns

The full data set `train_df` 64 features. The target column is labeled as `prognosis`.

### 13.3.2 Holdout Train and Test Data

We split out a hold-out test set (25% of the data) so we can calculate an example MAP@K

    import numpy as np
    from sklearn.model_selection import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(train_df.drop(target_column, axis=1), train_df[target_column],
                                                        random_state=42,
                                                        test_size=0.25,
                                                        stratify=train_df[target_column])
    train = pd.DataFrame(np.hstack((X_train, np.array(y_train).reshape(-1, 1))))
    test = pd.DataFrame(np.hstack((X_test, np.array(y_test).reshape(-1, 1))))
    train.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
    test.columns = [f"x{i}" for i in range(1, n_features+1)] + [target_column]
    print(train.shape)
    print(test.shape)
    train.head()

    (530, 65)
    (177, 65)

| x1 | x2 | x3 | x4 | x5 | x6 | x7 | x8 | x9 | x10 | ... | x56 | x57 | x58 | x59 | x60 | x61 | x62 | x63 | x64 | prognosis
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 2.0
1 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 4.0
2 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0
3 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 6.0
4 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 5.0

5 rows × 65 columns

    # add the dataset to the fun_control
    fun_control.update({"data": train_df, # full dataset,
                   "train": train,
                   "test": test,
                   "n_samples": n_samples,
                   "target_column": target_column})

## 13.4 Step 4: Specification of the Preprocessing Model

Data preprocesssing can be very simple, e.g., you can ignore it. Then you would choose the `prep_model` “None”:

    prep_model = None
    fun_control.update({"prep_model": prep_model})

A default approach for numerical data is the `StandardScaler` (mean 0, variance 1). This can be selected as follows:

    # prep_model = StandardScaler()
    # fun_control.update({"prep_model": prep_model})

Even more complicated pre-processing steps are possible, e.g., the follwing pipeline:

    # categorical_columns = []
    # one_hot_encoder = OneHotEncoder(handle_unknown="ignore", sparse_output=False)
    # prep_model = ColumnTransformer(
    #         transformers=[
    #             ("categorical", one_hot_encoder, categorical_columns),
    #         ],
    #         remainder=StandardScaler(),
    #     )

## 13.5 Step 5: Select Model (`algorithm`) and `core_model_hyper_dict`

The selection of the algorithm (ML model) that should be tuned is done by specifying the its name from the `sklearn` implementation. For example, the `SVC` support vector machine classifier is selected as follows:

`add_core_model_to_fun_control(SVC, fun_control, SklearnHyperDict)`

Other core_models are, e.g.,:

- RidgeCV
- GradientBoostingRegressor
- ElasticNet
- RandomForestClassifier
- LogisticRegression
- KNeighborsClassifier
- RandomForestClassifier
- GradientBoostingClassifier
- HistGradientBoostingClassifier

We will use the `RandomForestClassifier` classifier in this example.

    from sklearn.linear_model import RidgeCV
    from sklearn.ensemble import RandomForestClassifier
    from sklearn.svm import SVC
    from sklearn.linear_model import LogisticRegression
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.ensemble import GradientBoostingClassifier
    from sklearn.ensemble import GradientBoostingRegressor
    from sklearn.linear_model import ElasticNet
    from spotPython.hyperparameters.values import add_core_model_to_fun_control
    from spotPython.data.sklearn_hyper_dict import SklearnHyperDict
    from spotPython.fun.hypersklearn import HyperSklearn

    # core_model  = RidgeCV
    # core_model = GradientBoostingRegressor
    # core_model = ElasticNet
    core_model = RandomForestClassifier
    # core_model = SVC
    # core_model = LogisticRegression
    # core_model = KNeighborsClassifier
    # core_model = GradientBoostingClassifier
    add_core_model_to_fun_control(core_model=core_model,
                                  fun_control=fun_control,
                                  hyper_dict=SklearnHyperDict,
                                  filename=None)

Now `fun_control` has the information from the JSON file. The available hyperparameters are:

    print(*fun_control["core_model_hyper_dict"].keys(), sep="\n")

    n_estimators
    criterion
    max_depth
    min_samples_split
    min_samples_leaf
    min_weight_fraction_leaf
    max_features
    max_leaf_nodes
    min_impurity_decrease
    bootstrap
    oob_score

## 13.6 Step 6: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

### 13.6.1 Modify hyperparameter of type numeric and integer (boolean)

Numeric and boolean values can be modified using the `modify_hyper_parameter_bounds` method. For example, to change the `tol` hyperparameter of the `SVC` model to the interval [1e-3, 1e-2], the following code can be used:

`modify_hyper_parameter_bounds(fun_control, "tol", bounds=[1e-3, 1e-2])`

    from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
    # modify_hyper_parameter_bounds(fun_control, "tol", bounds=[1e-3, 1e-2])

### 13.6.2 Modify hyperparameter of type factor

`spotPython` provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 12.6.

Factors can be modified with the `modify_hyper_parameter_levels` function. For example, to exclude the `sigmoid` kernel from the tuning, the `kernel` hyperparameter of the `SVC` model can be modified as follows:

`modify_hyper_parameter_levels(fun_control, "kernel", ["linear", "rbf"])`

The new setting can be controlled via:

`fun_control["core_model_hyper_dict"]["kernel"]`

    from spotPython.hyperparameters.values import modify_hyper_parameter_levels
    # XGBoost:
    # modify_hyper_parameter_levels(fun_control, "loss", ["log_loss"])

Note: RandomForestClassifier and Out-of-bag Estimation

Since `oob_score` requires the `bootstrap` hyperparameter to `True`, we set the `oob_score` parameter to `False`. The `oob_score` is later discussed in Section 13.7.3.

    modify_hyper_parameter_bounds(fun_control, "bootstrap", bounds=[0, 1])
    modify_hyper_parameter_bounds(fun_control, "oob_score", bounds=[0, 0])

### 13.6.3 Optimizers

Optimizers are described in Section 12.6.1.

### 13.6.4 Selection of the Objective: Metric and Loss Functions

- Machine learning models are optimized with respect to a metric, for example, the `accuracy` function.
- Deep learning, e.g., neural networks are optimized with respect to a loss function, for example, the `cross_entropy` function and evaluated with respect to a metric, for example, the `accuracy` function.

## 13.7 Step 7: Selection of the Objective (Loss) Function

The loss function, that is usually used in deep learning for optimizing the weights of the net, is stored in the `fun_control` dictionary as `"loss_function"`.

### 13.7.1 Metric Function

There are two different types of metrics in `spotPython`:

1. `"metric_river"` is used for the river based evaluation via `eval_oml_iter_progressive`.
2. `"metric_sklearn"` is used for the sklearn based evaluation.

We will consider multi-class classification metrics, e.g., `mapk_score` and `top_k_accuracy_score`.

Predict Probabilities

In this multi-class classification example the machine learning algorithm should return the probabilities of the specific classes (`"predict_proba"`) instead of the predicted values.

We set `"predict_proba"` to `True` in the `fun_control` dictionary.

#### 13.7.1.1 The MAPK Metric

To select the MAPK metric, the following two entries can be added to the `fun_control` dictionary:

`"metric_sklearn": mapk_score"`

`"metric_params": {"k": 3}`.

#### 13.7.1.2 Other Metrics

Alternatively, other metrics for multi-class classification can be used, e.g.,: * top_k_accuracy_score or * roc_auc_score

The metric `roc_auc_score` requires the parameter `"multi_class"`, e.g.,

`"multi_class": "ovr"`.

This is set in the `fun_control` dictionary.

Weights

`spotPython` performs a minimization, therefore, metrics that should be maximized have to be multiplied by -1. This is done by setting `"weights"` to `-1`.

- The complete setup for the metric in our example is:

    from spotPython.utils.metrics import mapk_score
    fun_control.update({
                   "weights": -1,
                   "metric_sklearn": mapk_score,
                   "predict_proba": True,
                   "metric_params": {"k": 3},
                   })

### 13.7.2 Evaluation on Hold-out Data

- The default method for computing the performance is `"eval_holdout"`.
- Alternatively, cross-validation can be used for every machine learning model.
- Specifically for RandomForests, the OOB-score can be used.

    fun_control.update({
        "eval": "train_hold_out",
    })

### 13.7.3 OOB Score

Using the OOB-Score is a very efficient way to estimate the performance of a random forest classifier. The OOB-Score is calculated on the training data and does not require a hold-out test set. If the OOB-Score is used, the key “eval” in the `fun_control` dictionary should be set to `"oob_score"` as shown below.

OOB-Score

In addition to setting the key `"eval"` in the `fun_control` dictionary to `"oob_score"`, the keys `"oob_score"` and `"bootstrap"` have to be set to `True`, because the OOB-Score requires the bootstrap method.

- Uncomment the following lines to use the OOB-Score:

    fun_control.update({
        "eval": "eval_oob_score",
    })
    modify_hyper_parameter_bounds(fun_control, "bootstrap", bounds=[1, 1])
    modify_hyper_parameter_bounds(fun_control, "oob_score", bounds=[1, 1])

#### 13.7.3.1 Cross Validation

Instead of using the OOB-score, the classical cross validation can be used. The number of folds is set by the key `"k_folds"`. For example, to use 5-fold cross validation, the key `"k_folds"` is set to `5`. Uncomment the following line to use cross validation:

    # fun_control.update({
    #      "eval": "train_cv",
    #      "k_folds": 10,
    # })

## 13.8 Step 8: Calling the SPOT Function

### 13.8.1 Preparing the SPOT Call

- Get types and variable names as well as lower and upper bounds for the hyperparameters.

    # extract the variable types, names, and bounds
    from spotPython.hyperparameters.values import (get_bound_values,
        get_var_name,
        get_var_type,)
    var_type = get_var_type(fun_control)
    var_name = get_var_name(fun_control)
    fun_control.update({"var_type": var_type,
                        "var_name": var_name})
    lower = get_bound_values(fun_control, "lower")
    upper = get_bound_values(fun_control, "upper")

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control))

    | name                     | type   | default   |   lower |   upper | transform              |
    |--------------------------|--------|-----------|---------|---------|------------------------|
    | n_estimators             | int    | 7         |       5 |   10    | transform_power_2_int  |
    | criterion                | factor | gini      |       0 |    2    | None                   |
    | max_depth                | int    | 10        |       1 |   20    | transform_power_2_int  |
    | min_samples_split        | int    | 2         |       2 |  100    | None                   |
    | min_samples_leaf         | int    | 1         |       1 |   25    | None                   |
    | min_weight_fraction_leaf | float  | 0.0       |       0 |    0.01 | None                   |
    | max_features             | factor | sqrt      |       0 |    1    | transform_none_to_None |
    | max_leaf_nodes           | int    | 10        |       7 |   12    | transform_power_2_int  |
    | min_impurity_decrease    | float  | 0.0       |       0 |    0.01 | None                   |
    | bootstrap                | factor | 1         |       1 |    1    | None                   |
    | oob_score                | factor | 0         |       1 |    1    | None                   |

### 13.8.2 The Objective Function

The objective function is selected next. It implements an interface from `sklearn`’s training, validation, and testing methods to `spotPython`.

    from spotPython.fun.hypersklearn import HyperSklearn
    fun = HyperSklearn().fun_sklearn

### 13.8.3 Run the `Spot` Optimizer

- Run SPOT for approx. x mins (`max_time`).
- Note: the run takes longer, because the evaluation time of initial design (here: `initi_size`, 20 points) is not considered.

    from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
    hyper_dict=SklearnHyperDict().load()
    X_start = get_default_hyperparameters_as_array(fun_control, hyper_dict)
    X_start

    array([[ 7.,  0., 10.,  2.,  1.,  0.,  0., 10.,  0.,  1.,  0.]])

    import numpy as np
    from spotPython.spot import spot
    from math import inf
    spot_tuner = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = inf,
                       fun_repeats = 1,
                       max_time = MAX_TIME,
                       noise = False,
                       tolerance_x = np.sqrt(np.spacing(1)),
                       var_type = var_type,
                       var_name = var_name,
                       infill_criterion = "y",
                       n_points = 1,
                       seed=123,
                       log_level = 50,
                       show_models= False,
                       show_progress= True,
                       fun_control = fun_control,
                       design_control={"init_size": INIT_SIZE,
                                       "repeats": 1},
                       surrogate_control={"noise": True,
                                          "cod_type": "norm",
                                          "min_theta": -4,
                                          "max_theta": 3,
                                          "n_theta": len(var_name),
                                          "model_fun_evals": 10_000,
                                          "log_level": 50
                                          })
    spot_tuner.run(X_start=X_start)

    spotPython tuning: -0.34276729559748426 [----------] 1.42%

    spotPython tuning: -0.34276729559748426 [----------] 1.89%

    spotPython tuning: -0.34276729559748426 [----------] 2.47%

    spotPython tuning: -0.34276729559748426 [----------] 3.00%

    spotPython tuning: -0.34276729559748426 [----------] 3.39%

    spotPython tuning: -0.35062893081761004 [----------] 4.06%

    spotPython tuning: -0.35062893081761004 [----------] 4.83%

    spotPython tuning: -0.35062893081761004 [#---------] 5.63%

    spotPython tuning: -0.35062893081761004 [#---------] 6.93%

    spotPython tuning: -0.35062893081761004 [#---------] 7.59%

    spotPython tuning: -0.35062893081761004 [#---------] 8.89%

    spotPython tuning: -0.35062893081761004 [#---------] 9.85%

    spotPython tuning: -0.35062893081761004 [#---------] 12.72%

    spotPython tuning: -0.35062893081761004 [##--------] 17.63%

    spotPython tuning: -0.35062893081761004 [##--------] 19.90%

    spotPython tuning: -0.35062893081761004 [##--------] 23.24%

    spotPython tuning: -0.35062893081761004 [###-------] 26.55%

    spotPython tuning: -0.35188679245283017 [###-------] 29.94%

    spotPython tuning: -0.35188679245283017 [###-------] 34.01%

    spotPython tuning: -0.35188679245283017 [####------] 39.01%

    spotPython tuning: -0.35188679245283017 [####------] 43.06%

    spotPython tuning: -0.35188679245283017 [#####-----] 47.28%

    spotPython tuning: -0.35754716981132073 [#####-----] 51.26%

    spotPython tuning: -0.35754716981132073 [######----] 55.26%

    spotPython tuning: -0.35754716981132073 [######----] 59.51%

    spotPython tuning: -0.35754716981132073 [######----] 63.87%

    spotPython tuning: -0.35754716981132073 [#######---] 68.70%

    spotPython tuning: -0.35754716981132073 [########--] 75.57%

    spotPython tuning: -0.35754716981132073 [########--] 82.28%

    spotPython tuning: -0.35754716981132073 [#########-] 89.33%

    spotPython tuning: -0.3591194968553459 [##########] 96.08%

    spotPython tuning: -0.3591194968553459 [##########] 100.00% Done...

    <spotPython.spot.spot.Spot at 0x16d1c7af0>

## 13.9 Step 9: Tensorboard

The textual output shown in the console (or code cell) can be visualized with Tensorboard as described in Section 12.9, see also the description in the documentation: Tensorboard.

## 13.10 Step 10: Results

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.

    spot_tuner.plot_progress(log_y=False,
        filename="./figures/" + experiment_name+"_progress.png")

Progress plot. Black dots denote results from the initial design. Red dots illustrate the improvement found by the surrogate model based optimization.

- Print the results

    print(gen_design_table(fun_control=fun_control,
        spot=spot_tuner))

    | name                     | type   | default   |   lower |   upper |                 tuned | transform              |   importance | stars   |
    |--------------------------|--------|-----------|---------|---------|-----------------------|------------------------|--------------|---------|
    | n_estimators             | int    | 7         |     5.0 |    10.0 |                   9.0 | transform_power_2_int  |        38.85 | *       |
    | criterion                | factor | gini      |     0.0 |     2.0 |                   1.0 | None                   |       100.00 | ***     |
    | max_depth                | int    | 10        |     1.0 |    20.0 |                  12.0 | transform_power_2_int  |         0.01 |         |
    | min_samples_split        | int    | 2         |     2.0 |   100.0 |                  11.0 | None                   |        13.42 | *       |
    | min_samples_leaf         | int    | 1         |     1.0 |    25.0 |                   1.0 | None                   |         2.54 | *       |
    | min_weight_fraction_leaf | float  | 0.0       |     0.0 |    0.01 | 0.0059164181306543325 | None                   |         0.00 |         |
    | max_features             | factor | sqrt      |     0.0 |     1.0 |                   0.0 | transform_none_to_None |         0.02 |         |
    | max_leaf_nodes           | int    | 10        |     7.0 |    12.0 |                   9.0 | transform_power_2_int  |         2.14 | *       |
    | min_impurity_decrease    | float  | 0.0       |     0.0 |    0.01 |                   0.0 | None                   |         0.00 |         |
    | bootstrap                | factor | 1         |     1.0 |     1.0 |                   1.0 | None                   |         0.00 |         |
    | oob_score                | factor | 0         |     1.0 |     1.0 |                   1.0 | None                   |         0.00 |         |

### 13.10.1 Show variable importance

    spot_tuner.plot_importance(threshold=0.025, filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

### 13.10.2 Get Default Hyperparameters

    from spotPython.hyperparameters.values import get_default_values, transform_hyper_parameter_values
    values_default = get_default_values(fun_control)
    values_default = transform_hyper_parameter_values(fun_control=fun_control, hyper_parameter_values=values_default)
    values_default

    {'n_estimators': 128,
     'criterion': 'gini',
     'max_depth': 1024,
     'min_samples_split': 2,
     'min_samples_leaf': 1,
     'min_weight_fraction_leaf': 0.0,
     'max_features': 'sqrt',
     'max_leaf_nodes': 1024,
     'min_impurity_decrease': 0.0,
     'bootstrap': 1,
     'oob_score': 0}

    from sklearn.pipeline import make_pipeline
    model_default = make_pipeline(fun_control["prep_model"], fun_control["core_model"](**values_default))
    model_default

    Pipeline(steps=[('nonetype', None),
                    ('randomforestclassifier',
                     RandomForestClassifier(bootstrap=1, max_depth=1024,
                                            max_leaf_nodes=1024, n_estimators=128,
                                            oob_score=0))])

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
Pipeline

    Pipeline(steps=[('nonetype', None),
                    ('randomforestclassifier',
                     RandomForestClassifier(bootstrap=1, max_depth=1024,
                                            max_leaf_nodes=1024, n_estimators=128,
                                            oob_score=0))])

None

    None

RandomForestClassifier

    RandomForestClassifier(bootstrap=1, max_depth=1024, max_leaf_nodes=1024,
                           n_estimators=128, oob_score=0)

### 13.10.3 Get SPOT Results

    X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
    print(X)

    [[9.00000000e+00 1.00000000e+00 1.20000000e+01 1.10000000e+01
      1.00000000e+00 5.91641813e-03 0.00000000e+00 9.00000000e+00
      0.00000000e+00 1.00000000e+00 1.00000000e+00]]

    from spotPython.hyperparameters.values import assign_values, return_conf_list_from_var_dict
    v_dict = assign_values(X, fun_control["var_name"])
    return_conf_list_from_var_dict(var_dict=v_dict, fun_control=fun_control)

    [{'n_estimators': 512,
      'criterion': 'entropy',
      'max_depth': 4096,
      'min_samples_split': 11,
      'min_samples_leaf': 1,
      'min_weight_fraction_leaf': 0.0059164181306543325,
      'max_features': 'sqrt',
      'max_leaf_nodes': 512,
      'min_impurity_decrease': 0.0,
      'bootstrap': 1,
      'oob_score': 1}]

    from spotPython.hyperparameters.values import get_one_sklearn_model_from_X
    model_spot = get_one_sklearn_model_from_X(X, fun_control)
    model_spot

    RandomForestClassifier(bootstrap=1, criterion='entropy', max_depth=4096,
                           max_leaf_nodes=512, min_samples_split=11,
                           min_weight_fraction_leaf=0.0059164181306543325,
                           n_estimators=512, oob_score=1)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
RandomForestClassifier

    RandomForestClassifier(bootstrap=1, criterion='entropy', max_depth=4096,
                           max_leaf_nodes=512, min_samples_split=11,
                           min_weight_fraction_leaf=0.0059164181306543325,
                           n_estimators=512, oob_score=1)

### 13.10.4 Evaluate SPOT Results

- Fetch the data.

    from spotPython.utils.convert import get_Xy_from_df
    X_train, y_train = get_Xy_from_df(fun_control["train"], fun_control["target_column"])
    X_test, y_test = get_Xy_from_df(fun_control["test"], fun_control["target_column"])
    X_test.shape, y_test.shape

    ((177, 64), (177,))

- Fit the model with the tuned hyperparameters. This gives one result:

    model_spot.fit(X_train, y_train)
    y_pred = model_spot.predict_proba(X_test)
    res = mapk_score(y_true=y_test, y_pred=y_pred, k=3)
    res

    0.3615819209039548

    def repeated_eval(n, model):
        res_values = []
        for i in range(n):
            model.fit(X_train, y_train)
            y_pred = model.predict_proba(X_test)
            res = mapk_score(y_true=y_test, y_pred=y_pred, k=3)
            res_values.append(res)
        mean_res = np.mean(res_values)
        print(f"mean_res: {mean_res}")
        std_res = np.std(res_values)
        print(f"std_res: {std_res}")
        min_res = np.min(res_values)
        print(f"min_res: {min_res}")
        max_res = np.max(res_values)
        print(f"max_res: {max_res}")
        median_res = np.median(res_values)
        print(f"median_res: {median_res}")
        return mean_res, std_res, min_res, max_res, median_res

### 13.10.5 Handling Non-deterministic Results

- Because the model is non-determinstic, we perform \(n=30\) runs and calculate the mean and standard deviation of the performance metric.

    _ = repeated_eval(30, model_spot)

    mean_res: 0.3583804143126177
    std_res: 0.007006898349060751
    min_res: 0.34463276836158196
    max_res: 0.3757062146892655
    median_res: 0.3578154425612053

### 13.10.6 Evalution of the Default Hyperparameters

    model_default.fit(X_train, y_train)["randomforestclassifier"]

    RandomForestClassifier(bootstrap=1, max_depth=1024, max_leaf_nodes=1024,
                           n_estimators=128, oob_score=0)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
RandomForestClassifier

    RandomForestClassifier(bootstrap=1, max_depth=1024, max_leaf_nodes=1024,
                           n_estimators=128, oob_score=0)

- One evaluation of the default hyperparameters is performed on the hold-out test set.

    y_pred = model_default.predict_proba(X_test)
    mapk_score(y_true=y_test, y_pred=y_pred, k=3)

    0.3436911487758945

Since one single evaluation is not meaningful, we perform, similar to the evaluation of the SPOT results, \(n=30\) runs of the default setting and and calculate the mean and standard deviation of the performance metric.

    _ = repeated_eval(30, model_default)

    mean_res: 0.34497802887633394
    std_res: 0.01267398819635564
    min_res: 0.3229755178907721
    max_res: 0.36817325800376643
    median_res: 0.3427495291902071

### 13.10.7 Plot: Compare Predictions

    from spotPython.plot.validation import plot_confusion_matrix
    plot_confusion_matrix(model_default, fun_control, title = "Default")

    plot_confusion_matrix(model_spot, fun_control, title="SPOT")

    min(spot_tuner.y), max(spot_tuner.y)

    (-0.3591194968553459, -0.28962264150943395)

### 13.10.8 Cross-validated Evaluations

    from spotPython.sklearn.traintest import evaluate_cv
    fun_control.update({
         "eval": "train_cv",
         "k_folds": 10,
    })
    evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

    (0.359433962264151, None)

    fun_control.update({
         "eval": "test_cv",
         "k_folds": 10,
    })
    evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

    (0.31290849673202614, None)

- This is the evaluation that will be used in the comparison:

    fun_control.update({
         "eval": "data_cv",
         "k_folds": 10,
    })
    evaluate_cv(model=model_spot, fun_control=fun_control, verbose=0)

    (0.3648658618376928, None)

### 13.10.9 Detailed Hyperparameter Plots

    filename = "./figures/" + experiment_name
    spot_tuner.plot_important_hyperparameter_contour(filename=filename)

    n_estimators:  38.85223485225775
    criterion:  100.0
    min_samples_split:  13.424756161824943
    min_samples_leaf:  2.5425186088124936
    max_leaf_nodes:  2.1411547457633917

### 13.10.10 Parallel Coordinates Plot

    spot_tuner.parallel_plot()

### 13.10.11 Plot all Combinations of Hyperparameters

- Warning: this may take a while.

    PLOT_ALL = False
    if PLOT_ALL:
        n = spot_tuner.k
        for i in range(n-1):
            for j in range(i+1, n):
                spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)