# 12 HPT: PyTorch With `spotPython` and Ray Tune on CIFAR10

In this tutorial, we will show how `spotPython` can be integrated into the `PyTorch` training workflow. It is based on the tutorial “Hyperparameter Tuning with Ray Tune” from the `PyTorch` documentation (PyTorch 2023a), which is an extension of the tutorial “Training a Classifier” (PyTorch 2023b) for training a CIFAR10 image classifier.

This document refers to the following software versions:

- `python`: 3.10.10
- `torch`: 2.0.1
- `torchvision`: 0.15.0

    pip list | grep  "spot[RiverPython]"

    spotPython                 0.3.0
    spotRiver                  0.0.93

    Note: you may need to restart the kernel to use updated packages.

`spotPython` can be installed via pip1.

    !pip install spotPython

- Uncomment the following lines if you want to for (re-)installation the latest version of `spotPython` from gitHub.

    # import sys
    # !{sys.executable} -m pip install --upgrade build
    # !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

Results that refer to the `Ray Tune` package are taken from https://PyTorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html 2.

## 12.1 Step 1: Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.

Caution: Run time and initial design size should be increased for real experiments

- MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
- INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.

Note: Device selection

- The device can be selected by setting the variable `DEVICE`.
- Since we are using a simple neural net, the setting `"cpu"` is preferred (on Mac).
- If you have a GPU, you can use `"cuda:0"` instead.
- If DEVICE is set to `None`, `spotPython` will automatically select the device.
  - This might result in `"mps"` on Macs, which is not the best choice for simple neural nets.

    MAX_TIME = 10
    INIT_SIZE = 5
    DEVICE = "cpu" # "cuda:0"

    from spotPython.utils.device import getDevice
    DEVICE = getDevice(DEVICE)
    print(DEVICE)

    cpu

    import os
    import copy
    import socket
    import warnings
    from datetime import datetime
    from dateutil.tz import tzlocal
    start_time = datetime.now(tzlocal())
    HOSTNAME = socket.gethostname().split(".")[0]
    experiment_name = '14-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
    experiment_name = experiment_name.replace(':', '-')
    print(experiment_name)
    if not os.path.exists('./figures'):
        os.makedirs('./figures')
    warnings.filterwarnings("ignore")

    14-torch_bartz09_10min_5init_2023-07-04_01-03-35

## 12.2 Step 2: Initialization of the `fun_control` Dictionary

`spotPython` uses a Python dictionary for storing the information required for the hyperparameter tuning process. This dictionary is called `fun_control` and is initialized with the function `fun_control_init`. The function `fun_control_init` returns a skeleton dictionary. The dictionary is filled with the required information for the hyperparameter tuning process. It stores the hyperparameter tuning settings, e.g., the deep learning network architecture that should be tuned, the classification (or regression) problem, and the data that is used for the tuning. The dictionary is used as an input for the SPOT function.

Caution: Tensorboard does not work under Windows

- Since tensorboard does not work under Windows, we recommend setting the parameter `tensorboard_path` to `None` if you are working under Windows.

    from spotPython.utils.init import fun_control_init
    fun_control = fun_control_init(task="classification",
        tensorboard_path="runs/14_spot_ray_hpt_torch_cifar10",
        device=DEVICE,)

## 12.3 Step 3: PyTorch Data Loading

The data loading process is implemented in the same manner as described in the Section “Data loaders” in PyTorch (2023a). The data loaders are wrapped into the function `load_data_cifar10` which is identical to the function `load_data` in PyTorch (2023a). A global data directory is used, which allows sharing the data directory between different trials. The method `load_data_cifar10` is part of the `spotPython` package and can be imported from `spotPython.data.torchdata`.

In the following step, the test and train data are added to the dictionary `fun_control`.

    from spotPython.data.torchdata import load_data_cifar10
    train, test = load_data_cifar10()
    n_samples = len(train)
    # add the dataset to the fun_control
    fun_control.update({
        "train": train,
        "test": test,
        "n_samples": n_samples})

    Files already downloaded and verified

    Files already downloaded and verified

## 12.4 Step 4: Specification of the Preprocessing Model

After the training and test data are specified and added to the `fun_control` dictionary, `spotPython` allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables. The preprocessing model is called `prep_model` (“preparation” or pre-processing) and includes steps that are not subject to the hyperparameter tuning process. The preprocessing model is specified in the `fun_control` dictionary. The preprocessing model can be implemented as a `sklearn` pipeline. The following code shows a typical preprocessing pipeline:

    categorical_columns = ["cities", "colors"]
    one_hot_encoder = OneHotEncoder(handle_unknown="ignore",
                                        sparse_output=False)
    prep_model = ColumnTransformer(
            transformers=[
                 ("categorical", one_hot_encoder, categorical_columns),
             ],
             remainder=StandardScaler(),
         )

Because the Ray Tune (`ray[tune]`) hyperparameter tuning as described in PyTorch (2023a) does not use a preprocessing model, the preprocessing model is set to `None` here.

    prep_model = None
    fun_control.update({"prep_model": prep_model})

## 12.5 Step 5: Select Model (`algorithm`) and `core_model_hyper_dict`

The same neural network model as implemented in the section “Configurable neural network” of the `PyTorch` tutorial (PyTorch 2023a) is used here. We will show the implementation from PyTorch (2023a) in Section 12.5.0.1 first, before the extended implementation with `spotPython` is shown in Section 12.5.0.2.

#### 12.5.0.1 Implementing a Configurable Neural Network With Ray Tune

We used the same hyperparameters that are implemented as configurable in the `PyTorch` tutorial. We specify the layer sizes, namely `l1` and `l2`, of the fully connected layers:

    class Net(nn.Module):
        def __init__(self, l1=120, l2=84):
            super(Net, self).__init__()
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, l1)
            self.fc2 = nn.Linear(l1, l2)
            self.fc3 = nn.Linear(l2, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1, 16 * 5 * 5)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

The learning rate, i.e., `lr`, of the optimizer is made configurable, too:

    optimizer = optim.SGD(net.parameters(), lr=config["lr"], momentum=0.9)

#### 12.5.0.2 Implementing a Configurable Neural Network With spotPython

`spotPython` implements a class which is similar to the class described in the `PyTorch` tutorial. The class is called `Net_CIFAR10` and is implemented in the file `netcifar10.py`.

    from torch import nn
    import torch.nn.functional as F
    import spotPython.torch.netcore as netcore

    class Net_CIFAR10(netcore.Net_Core):
        def __init__(self, l1, l2, lr_mult, batch_size, epochs, k_folds, patience,
        optimizer, sgd_momentum):
            super(Net_CIFAR10, self).__init__(
                lr_mult=lr_mult,
                batch_size=batch_size,
                epochs=epochs,
                k_folds=k_folds,
                patience=patience,
                optimizer=optimizer,
                sgd_momentum=sgd_momentum,
            )
            self.conv1 = nn.Conv2d(3, 6, 5)
            self.pool = nn.MaxPool2d(2, 2)
            self.conv2 = nn.Conv2d(6, 16, 5)
            self.fc1 = nn.Linear(16 * 5 * 5, l1)
            self.fc2 = nn.Linear(l1, l2)
            self.fc3 = nn.Linear(l2, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.conv1(x)))
            x = self.pool(F.relu(self.conv2(x)))
            x = x.view(-1, 16 * 5 * 5)
            x = F.relu(self.fc1(x))
            x = F.relu(self.fc2(x))
            x = self.fc3(x)
            return x

### 12.5.1 The `Net_Core` class

`Net_CIFAR10` inherits from the class `Net_Core` which is implemented in the file `netcore.py`. It implements the additional attributes that are common to all neural network models. The `Net_Core` class is implemented in the file `netcore.py`. It implements hyperparameters as attributes, that are not used by the `core_model`, e.g.:

- optimizer (`optimizer`),
- learning rate (`lr`),
- batch size (`batch_size`),
- epochs (`epochs`),
- k_folds (`k_folds`), and
- early stopping criterion “patience” (`patience`).

Users can add further attributes to the class. The class `Net_Core` is shown below.

    from torch import nn

    class Net_Core(nn.Module):
        def __init__(self, lr_mult, batch_size, epochs, k_folds, patience,
            optimizer, sgd_momentum):
            super(Net_Core, self).__init__()
            self.lr_mult = lr_mult
            self.batch_size = batch_size
            self.epochs = epochs
            self.k_folds = k_folds
            self.patience = patience
            self.optimizer = optimizer
            self.sgd_momentum = sgd_momentum

### 12.5.2 Comparison of the Approach Described in the PyTorch Tutorial With spotPython

Comparing the class `Net` from the `PyTorch` tutorial and the class `Net_CIFAR10` from `spotPython`, we see that the class `Net_CIFAR10` has additional attributes and does not inherit from `nn` directly. It adds an additional class, `Net_core`, that takes care of additional attributes that are common to all neural network models, e.g., the learning rate multiplier `lr_mult` or the batch size `batch_size`.

`spotPython`’s `core_model` implements an instance of the `Net_CIFAR10` class. In addition to the basic neural network model, the `core_model` can use these additional attributes. `spotPython` provides methods for handling these additional attributes to guarantee 100% compatibility with the `PyTorch` classes. The method `add_core_model_to_fun_control` adds the hyperparameters and additional attributes to the `fun_control` dictionary. The method is shown below.

    from spotPython.torch.netcifar10 import Net_CIFAR10
    from spotPython.data.torch_hyper_dict import TorchHyperDict
    from spotPython.hyperparameters.values import add_core_model_to_fun_control
    core_model = Net_CIFAR10
    add_core_model_to_fun_control(core_model=core_model,
                                  fun_control=fun_control,
                                  hyper_dict=TorchHyperDict,
                                  filename=None)

### 12.5.3 The Search Space: Hyperparameters

In Section 12.5.4, we first describe how to configure the search space with `ray[tune]` (as shown in PyTorch (2023a)) and then how to configure the search space with `spotPython` in -14.

### 12.5.4 Configuring the Search Space With Ray Tune

Ray Tune’s search space can be configured as follows (PyTorch 2023a):

    config = {
        "l1": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
        "l2": tune.sample_from(lambda _: 2**np.random.randint(2, 9)),
        "lr": tune.loguniform(1e-4, 1e-1),
        "batch_size": tune.choice([2, 4, 8, 16])
    }

The `tune.sample_from()` function enables the user to define sample methods to obtain hyperparameters. In this example, the `l1` and `l2` parameters should be powers of 2 between 4 and 256, so either 4, 8, 16, 32, 64, 128, or 256. The `lr` (learning rate) should be uniformly sampled between 0.0001 and 0.1. Lastly, the batch size is a choice between 2, 4, 8, and 16.

At each trial, `ray[tune]` will randomly sample a combination of parameters from these search spaces. It will then train a number of models in parallel and find the best performing one among these. `ray[tune]` uses the `ASHAScheduler` which will terminate bad performing trials early.

### 12.5.5 Configuring the Search Space With spotPython

#### 12.5.5.1 The `hyper_dict` Hyperparameters for the Selected Algorithm

`spotPython` uses `JSON` files for the specification of the hyperparameters. Users can specify their individual `JSON` files, or they can use the `JSON` files provided by `spotPython`. The `JSON` file for the `core_model` is called `torch_hyper_dict.json`.

In contrast to `ray[tune]`, `spotPython` can handle numerical, boolean, and categorical hyperparameters. They can be specified in the `JSON` file in a similar way as the numerical hyperparameters as shown below. Each entry in the `JSON` file represents one hyperparameter with the following structure: `type`, `default`, `transform`, `lower`, and `upper`.

    "factor_hyperparameter": {
        "levels": ["A", "B", "C"],
        "type": "factor",
        "default": "B",
        "transform": "None",
        "core_model_parameter_type": "str",
        "lower": 0,
        "upper": 2},

The corresponding entries for the core_model` class are shown below.

    fun_control['core_model_hyper_dict']

    {'l1': {'type': 'int',
      'default': 5,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 9},
     'l2': {'type': 'int',
      'default': 5,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 9},
     'lr_mult': {'type': 'float',
      'default': 1.0,
      'transform': 'None',
      'lower': 0.1,
      'upper': 10.0},
     'batch_size': {'type': 'int',
      'default': 4,
      'transform': 'transform_power_2_int',
      'lower': 1,
      'upper': 4},
     'epochs': {'type': 'int',
      'default': 3,
      'transform': 'transform_power_2_int',
      'lower': 3,
      'upper': 4},
     'k_folds': {'type': 'int',
      'default': 1,
      'transform': 'None',
      'lower': 1,
      'upper': 1},
     'patience': {'type': 'int',
      'default': 5,
      'transform': 'None',
      'lower': 2,
      'upper': 10},
     'optimizer': {'levels': ['Adadelta',
       'Adagrad',
       'Adam',
       'AdamW',
       'SparseAdam',
       'Adamax',
       'ASGD',
       'NAdam',
       'RAdam',
       'RMSprop',
       'Rprop',
       'SGD'],
      'type': 'factor',
      'default': 'SGD',
      'transform': 'None',
      'class_name': 'torch.optim',
      'core_model_parameter_type': 'str',
      'lower': 0,
      'upper': 12},
     'sgd_momentum': {'type': 'float',
      'default': 0.0,
      'transform': 'None',
      'lower': 0.0,
      'upper': 1.0}}

## 12.6 Step 6: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

Ray tune (PyTorch 2023a) does not provide a way to change the specified hyperparameters without re-compilation. However, `spotPython` provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions are described in the following.

#### 12.6.0.1 Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

After specifying the model, the corresponding hyperparameters, their types and bounds are loaded from the `JSON` file `torch_hyper_dict.json`. After loading, the user can modify the hyperparameters, e.g., the bounds. `spotPython` provides a simple rule for de-activating hyperparameters: If the lower and the upper bound are set to identical values, the hyperparameter is de-activated. This is useful for the hyperparameter tuning, because it allows to specify a hyperparameter in the `JSON` file, but to de-activate it in the `fun_control` dictionary. This is done in the next step.

#### 12.6.0.2 Modify Hyperparameters of Type numeric and integer (boolean)

Since the hyperparameter `k_folds` is not used in the `PyTorch` tutorial, it is de-activated here by setting the lower and upper bound to the same value. Note, `k_folds` is of type “integer”.

    from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
    modify_hyper_parameter_bounds(fun_control,
        "batch_size", bounds=[1, 5])
    modify_hyper_parameter_bounds(fun_control,
        "k_folds", bounds=[0, 0])
    modify_hyper_parameter_bounds(fun_control,
        "patience", bounds=[3, 3])

#### 12.6.0.3 Modify Hyperparameter of Type factor

In a similar manner as for the numerical hyperparameters, the categorical hyperparameters can be modified. New configurations can be chosen by adding or deleting levels. For example, the hyperparameter `optimizer` can be re-configured as follows:

In the following setting, two optimizers (`"SGD"` and `"Adam"`) will be compared during the `spotPython` hyperparameter tuning. The hyperparameter `optimizer` is active.

    from spotPython.hyperparameters.values import modify_hyper_parameter_levels
    modify_hyper_parameter_levels(fun_control,
         "optimizer", ["SGD", "Adam"])

The hyperparameter `optimizer` can be de-activated by choosing only one value (level), here: `"SGD"`.

    modify_hyper_parameter_levels(fun_control, "optimizer", ["SGD"])

As discussed in Section 12.6.1, there are some issues with the LBFGS optimizer. Therefore, the usage of the LBFGS optimizer is not deactivated in `spotPython` by default. However, the LBFGS optimizer can be activated by adding it to the list of optimizers. `Rprop` was removed, because it does perform very poorly (as some pre-tests have shown). However, it can also be activated by adding it to the list of optimizers. Since `SparseAdam` does not support dense gradients, `Adam` was used instead. Therefore, there are 10 default optimizers:

    modify_hyper_parameter_levels(fun_control, "optimizer",
        ["Adadelta", "Adagrad", "Adam", "AdamW", "Adamax", "ASGD",
        "NAdam", "RAdam", "RMSprop", "SGD"])

### 12.6.1 Optimizers

Table 12.1 shows some of the optimizers available in `PyTorch`:

\(a\) denotes (0.9,0.999), \(b\) (0.5,1.2), and \(c\) (1e-6, 50), respectively. \(R\) denotes `required, but unspecified`. “m” denotes `momentum`, “w_d” `weight_decay`, “d” `dampening`, “n” `nesterov`, “r” `rho`, “l_s” `learning rate for scaling delta`, “l_d” `lr_decay`, “b” `betas`, “l” `lambd`, “a” `alpha`, “m_d” for `momentum_decay`, “e” `etas`, and “s_s” for `step_sizes`.

Table 12.1: Optimizers available in PyTorch (selection). The default values are shown in the table.
Optimizer | lr | m | w_d | d | n | r | l_s | l_d | b | l | a | m_d | e | s_s
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
Adadelta | - | - | 0. | - | - | 0.9 | 1. | - | - | - | - | - | - | -
Adagrad | 1e-2 | - | 0. | - | - | - | - | 0. | - | - | - | - | - | -
Adam | 1e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | -
AdamW | 1e-3 | - | 1e-2 | - | - | - | - | - | \(a\) | - | - | - | - | -
SparseAdam | 1e-3 | - | - | - | - | - | - | - | \(a\) | - | - | - | - | -
Adamax | 2e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | -
ASGD | 1e-2 | .9 | 0. | - | F | - | - | - | - | 1e-4 | .75 | - | - | -
LBFGS | 1. | - | - | - | - | - | - | - | - | - | - | - | - | -
NAdam | 2e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | 0 | - | -
RAdam | 1e-3 | - | 0. | - | - | - | - | - | \(a\) | - | - | - | - | -
RMSprop | 1e-2 | 0. | 0. | - | - | - | - | - | \(a\) | - | - | - | - | -
Rprop | 1e-2 | - | - | - | - | - | - | - | - | - | \(b\) | \(c\) | - | -
SGD | \(R\) | 0. | 0. | 0. | F | - | - | - | - | - | - | - | - | -

`spotPython` implements an `optimization` handler that maps the optimizer names to the corresponding `PyTorch` optimizers.

A note on LBFGS

We recommend deactivating `PyTorch`’s LBFGS optimizer, because it does not perform very well. The `PyTorch` documentation, see https://pytorch.org/docs/stable/generated/torch.optim.LBFGS.html#torch.optim.LBFGS, states:

> This is a very memory intensive optimizer (it requires additional `param_bytes * (history_size + 1)` bytes). If it doesn’t fit in memory try reducing the history size, or use a different algorithm.

Furthermore, the LBFGS optimizer is not compatible with the `PyTorch` tutorial. The reason is that the LBFGS optimizer requires the `closure` function, which is not implemented in the `PyTorch` tutorial. Therefore, the `LBFGS` optimizer is recommended here. Since there are ten optimizers in the portfolio, it is not recommended tuning the hyperparameters that effect one single optimizer only.

A note on the learning rate

`spotPython` provides a multiplier for the default learning rates, `lr_mult`, because optimizers use different learning rates. Using a multiplier for the learning rates might enable a simultaneous tuning of the learning rates for all optimizers. However, this is not recommended, because the learning rates are not comparable across optimizers. Therefore, we recommend fixing the learning rate for all optimizers if multiple optimizers are used. This can be done by setting the lower and upper bounds of the learning rate multiplier to the same value as shown below.

Thus, the learning rate, which affects the `SGD` optimizer, will be set to a fixed value. We choose the default value of `1e-3` for the learning rate, because it is used in other `PyTorch` examples (it is also the default value used by `spotPython` as defined in the `optimizer_handler()` method). We recommend tuning the learning rate later, when a reduced set of optimizers is fixed. Here, we will demonstrate how to select in a screening phase the optimizers that should be used for the hyperparameter tuning.

For the same reason, we will fix the `sgd_momentum` to `0.9`.

    modify_hyper_parameter_bounds(fun_control,
        "lr_mult", bounds=[1.0, 1.0])
    modify_hyper_parameter_bounds(fun_control,
        "sgd_momentum", bounds=[0.9, 0.9])

## 12.7 Step 7: Selection of the Objective (Loss) Function

### 12.7.1 Evaluation: Data Splitting

The evaluation procedure requires the specification of the way how the data is split into a train and a test set and the loss function (and a metric). As a default, `spotPython` provides a standard hold-out data split and cross validation.

### 12.7.2 Hold-out Data Split

If a hold-out data split is used, the data will be partitioned into a training, a validation, and a test data set. The split depends on the setting of the `eval` parameter. If `eval` is set to `train_hold_out`, one data set, usually the original training data set, is split into a new training and a validation data set. The training data set is used for training the model. The validation data set is used for the evaluation of the hyperparameter configuration and early stopping to prevent overfitting. In this case, the original test data set is not used.

Note

`spotPython` returns the hyperparameters of the machine learning and deep learning models, e.g., number of layers, learning rate, or optimizer, but not the model weights. Therefore, after the SPOT run is finished, the corresponding model with the optimized architecture has to be trained again with the best hyperparameter configuration. The training is performed on the training data set. The test data set is used for the final evaluation of the model.

Summarizing, the following splits are performed in the hold-out setting:

1. Run `spotPython` with `eval` set to `train_hold_out` to determine the best hyperparameter configuration.
2. Train the model with the best hyperparameter configuration (“architecture”) on the training data set: `train_tuned(model_spot, train, "model_spot.pt")`.
3. Test the model on the test data: `test_tuned(model_spot, test, "model_spot.pt")`

These steps will be exemplified in the following sections.

In addition to this `hold-out` setting, `spotPython` provides another hold-out setting, where an explicit test data is specified by the user that will be used as the validation set. To choose this option, the `eval` parameter is set to `test_hold_out`. In this case, the training data set is used for the model training. Then, the explicitly defined test data set is used for the evaluation of the hyperparameter configuration (the validation).

### 12.7.3 Cross-Validation

The cross validation setting is used by setting the `eval` parameter to `train_cv` or `test_cv`. In both cases, the data set is split into \(k\) folds. The model is trained on \(k-1\) folds and evaluated on the remaining fold. This is repeated \(k\) times, so that each fold is used exactly once for evaluation. The final evaluation is performed on the test data set. The cross validation setting is useful for small data sets, because it allows to use all data for training and evaluation. However, it is computationally expensive, because the model has to be trained \(k\) times.

Note

Combinations of the above settings are possible, e.g., cross validation can be used for training and hold-out for evaluation or vice versa. Also, cross validation can be used for training and testing. Because cross validation is not used in the `PyTorch` tutorial (PyTorch 2023a), it is not considered further here.

### 12.7.4 Overview of the Evaluation Settings

#### 12.7.4.1 Settings for the Hyperparameter Tuning

An overview of the training evaluations is shown in Table 12.2. `"train_cv"` and `"test_cv"` use `sklearn.model_selection.KFold()` internally. More details on the data splitting are provided in Section 18.14 (in the Appendix).

Table 12.2: Overview of the evaluation settings.
`eval` | `train` | `test` | function | comment
--- | --- | --- | --- | ---
`"train_hold_out"` | \(\checkmark\) |  | `train_one_epoch()`, `validate_one_epoch()` for early stopping | splits the `train` data set internally
`"test_hold_out"` | \(\checkmark\) | \(\checkmark\) | `train_one_epoch()`, `validate_one_epoch()` for early stopping | use the `test data set` for `validate_one_epoch()`
`"train_cv"` | \(\checkmark\) |  | `evaluate_cv(net, train)` | CV using the `train` data set
`"test_cv"` |  | \(\checkmark\) | `evaluate_cv(net, test)` | CV using the `test` data set . Identical to `"train_cv"`, uses only test data.

#### 12.7.4.2 Settings for the Final Evaluation of the Tuned Architecture

##### 12.7.4.2.1 Training of the Tuned Architecture

`train_tuned(model, train)`: train the model with the best hyperparameter configuration (or simply the default) on the training data set. It splits the `train`data into new `train` and `validation` sets using `create_train_val_data_loaders()`, which calls `torch.utils.data.random_split()` internally. Currently, 60% of the data is used for training and 40% for validation. The `train` data is used for training the model with `train_hold_out()`. The `validation` data is used for early stopping using `validate_fold_or_hold_out()` on the `validation` data set.

##### 12.7.4.2.2 Testing of the Tuned Architecture

`test_tuned(model, test)`: test the model on the test data set. No data splitting is performed. The (trained) model is evaluated using the `validate_fold_or_hold_out()` function. Note: During training, `"shuffle"` is set to `True`, whereas during testing, `"shuffle"` is set to `False`.

Section 18.14.1.4 describes the final evaluation of the tuned architecture.

    fun_control.update({
        "eval": "train_hold_out",
        "path": "torch_model.pt",
        "shuffle": True})

### 12.7.5 Evaluation: Loss Functions and Metrics

The key `"loss_function"` specifies the loss function which is used during the optimization. There are several different loss functions under `PyTorch`’s `nn` package. For example, a simple loss is `MSELoss`, which computes the mean-squared error between the output and the target. In this tutorial we will use `CrossEntropyLoss`, because it is also used in the `PyTorch` tutorial.

    from torch.nn import CrossEntropyLoss
    loss_function = CrossEntropyLoss()
    fun_control.update({"loss_function": loss_function})

In addition to the loss functions, `spotPython` provides access to a large number of metrics.

- The key `"metric_sklearn"` is used for metrics that follow the `scikit-learn` conventions.
- The key `"river_metric"` is used for the river based evaluation (Montiel et al. 2021) via `eval_oml_iter_progressive`, and
- the key `"metric_torch"` is used for the metrics from `TorchMetrics`.

`TorchMetrics` is a collection of more than 90 PyTorch metrics, see https://torchmetrics.readthedocs.io/en/latest/. Because the `PyTorch` tutorial uses the accuracy as metric, we use the same metric here. Currently, accuracy is computed in the tutorial’s example code. We will use `TorchMetrics` instead, because it offers more flexibilty, e.g., it can be used for regression and classification. Furthermore, `TorchMetrics` offers the following advantages:

    * A standardized interface to increase reproducibility
    * Reduces Boilerplate
    * Distributed-training compatible
    * Rigorously tested
    * Automatic accumulation over batches
    * Automatic synchronization between multiple devices

Therefore, we set

    import torchmetrics
    metric_torch = torchmetrics.Accuracy(task="multiclass", num_classes=10).to(fun_control["device"])
    fun_control.update({"metric_torch": metric_torch})

## 12.8 Step 8: Calling the SPOT Function

### 12.8.1 Preparing the SPOT Call

The following code passes the information about the parameter ranges and bounds to `spot`.

    from spotPython.hyperparameters.values import (
        get_var_type,
        get_var_name,
        get_bound_values
        )
    var_type = get_var_type(fun_control)
    var_name = get_var_name(fun_control)
    fun_control.update({"var_type": var_type,
                        "var_name": var_name})

    lower = get_bound_values(fun_control, "lower")
    upper = get_bound_values(fun_control, "upper")

Now, the dictionary `fun_control` contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method `gen_design_table` generates a design table as follows:

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control))

    | name         | type   | default   |   lower |   upper | transform             |
    |--------------|--------|-----------|---------|---------|-----------------------|
    | l1           | int    | 5         |     2   |     9   | transform_power_2_int |
    | l2           | int    | 5         |     2   |     9   | transform_power_2_int |
    | lr_mult      | float  | 1.0       |     1   |     1   | None                  |
    | batch_size   | int    | 4         |     1   |     5   | transform_power_2_int |
    | epochs       | int    | 3         |     3   |     4   | transform_power_2_int |
    | k_folds      | int    | 1         |     0   |     0   | None                  |
    | patience     | int    | 5         |     3   |     3   | None                  |
    | optimizer    | factor | SGD       |     0   |     9   | None                  |
    | sgd_momentum | float  | 0.0       |     0.9 |     0.9 | None                  |

This allows to check if all information is available and if the information is correct. ?tbl-design shows the experimental design for the hyperparameter tuning. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The transformation function is used to transform the hyperparameter values from the unit hypercube to the original domain. The transformation function is applied to the hyperparameter values before the evaluation of the objective function. Hyperparameter transformations are shown in the column “transform”, e.g., the `l1` default is `5`, which results in the value \(2^5 = 32\) for the network, because the transformation `transform_power_2_int` was selected in the `JSON` file. The default value of the `batch_size` is set to `4`, which results in a batch size of \(2^4 = 16\).

### 12.8.2 The Objective Function `fun_torch`

The objective function `fun_torch` is selected next. It implements an interface from `PyTorch`’s training, validation, and testing methods to `spotPython`.

    from spotPython.fun.hypertorch import HyperTorch
    fun = HyperTorch().fun_torch

### 12.8.3 Using Default Hyperparameters or Results from Previous Runs

We add the default setting to the initial design:

    from spotPython.hyperparameters.values import get_default_hyperparameters_as_array
    hyper_dict=TorchHyperDict().load()
    X_start = get_default_hyperparameters_as_array(fun_control, hyper_dict)

### 12.8.4 Starting the Hyperparameter Tuning

The `spotPython` hyperparameter tuning is started by calling the `Spot` function. Here, we will run the tuner for approximately 30 minutes (`max_time`). Note: the initial design is always evaluated in the `spotPython` run. As a consequence, the run may take longer than specified by `max_time`, because the evaluation time of initial design (here: `init_size`, 10 points) is performed independently of `max_time`. During the run, results from the training is shown. These results can be visualized with Tensorboard as will be shown in Section 12.9.

    from spotPython.spot import spot
    from math import inf
    import numpy as np
    spot_tuner = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = inf,
                       fun_repeats = 1,
                       max_time = MAX_TIME,
                       noise = False,
                       tolerance_x = np.sqrt(np.spacing(1)),
                       var_type = var_type,
                       var_name = var_name,
                       infill_criterion = "y",
                       n_points = 1,
                       seed=123,
                       log_level = 50,
                       show_models= False,
                       show_progress= True,
                       fun_control = fun_control,
                       design_control={"init_size": INIT_SIZE,
                                       "repeats": 1},
                       surrogate_control={"noise": True,
                                          "cod_type": "norm",
                                          "min_theta": -4,
                                          "max_theta": 3,
                                          "n_theta": len(var_name),
                                          "model_fun_evals": 10_000,
                                          "log_level": 50
                                          })
    spot_tuner.run(X_start=X_start)

    config: {'l1': 128, 'l2': 8, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.3822999894618988 | Loss: 1.6569919290542603 | Acc: 0.3823000000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.4539499878883362 | Loss: 1.4899173261642455 | Acc: 0.4539500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.4918999969959259 | Loss: 1.3893225684165955 | Acc: 0.4919000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5101500153541565 | Loss: 1.3411697679519654 | Acc: 0.5101500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.5455999970436096 | Loss: 1.2716289128303528 | Acc: 0.5456000000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5616000294685364 | Loss: 1.2248678792953491 | Acc: 0.5616000000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5623499751091003 | Loss: 1.2264307149887086 | Acc: 0.5623500000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.5799000263214111 | Loss: 1.1842727457046509 | Acc: 0.5799000000000000.
    Epoch: 9 |

    MulticlassAccuracy: 0.5939000248908997 | Loss: 1.1541707914352417 | Acc: 0.5939000000000000.
    Epoch: 10 |

    MulticlassAccuracy: 0.5910500288009644 | Loss: 1.1763400409698486 | Acc: 0.5910500000000000.
    Epoch: 11 |

    MulticlassAccuracy: 0.5954499840736389 | Loss: 1.1757343863487244 | Acc: 0.5954500000000000.
    Epoch: 12 |

    MulticlassAccuracy: 0.5922499895095825 | Loss: 1.1785970301628113 | Acc: 0.5922500000000001.
    Early stopping at epoch 11
    Returned to Spot: Validation loss: 1.1785970301628113

    config: {'l1': 16, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.4137000143527985 | Loss: 1.5771867493391036 | Acc: 0.4137000000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.4795500040054321 | Loss: 1.4186543140649797 | Acc: 0.4795500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.4794999957084656 | Loss: 1.4321544742107390 | Acc: 0.4795000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.4969500005245209 | Loss: 1.4049549581408500 | Acc: 0.4969500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.4853500127792358 | Loss: 1.4105496326088904 | Acc: 0.4853500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.4887500107288361 | Loss: 1.4176285049676896 | Acc: 0.4887500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.4918999969959259 | Loss: 1.4230854102253914 | Acc: 0.4919000000000000.
    Early stopping at epoch 6
    Returned to Spot: Validation loss: 1.4230854102253914

    config: {'l1': 256, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 2, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'RMSprop', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.0994499996304512 | Loss: 2.3137239639997484 | Acc: 0.0994500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.0993499979376793 | Loss: 2.3085385819911957 | Acc: 0.0993500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.0988000035285950 | Loss: 2.3045743632316591 | Acc: 0.0988000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.0966999977827072 | Loss: 2.3057950327873229 | Acc: 0.0967000000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.0988000035285950 | Loss: 2.3120626847267149 | Acc: 0.0988000000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.0966999977827072 | Loss: 2.3112036952495574 | Acc: 0.0967000000000000.
    Early stopping at epoch 5
    Returned to Spot: Validation loss: 2.3112036952495574

    config: {'l1': 8, 'l2': 32, 'lr_mult': 1.0, 'batch_size': 4, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.3768500089645386 | Loss: 1.6525698980569838 | Acc: 0.3768500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.4381000101566315 | Loss: 1.5250679602503776 | Acc: 0.4381000000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.4542500078678131 | Loss: 1.4748067722111939 | Acc: 0.4542500000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.4587000012397766 | Loss: 1.4901986456006766 | Acc: 0.4587000000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.4853999912738800 | Loss: 1.3857398863971233 | Acc: 0.4854000000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5037500262260437 | Loss: 1.3827529514372350 | Acc: 0.5037500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5183500051498413 | Loss: 1.3676455246344208 | Acc: 0.5183500000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.5340999960899353 | Loss: 1.3272761303044855 | Acc: 0.5341000000000000.
    Returned to Spot: Validation loss: 1.3272761303044855

    config: {'l1': 64, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adagrad', 'sgd_momentum': 0.9}
    Error in Net_Core. Call to evaluate_hold_out() failed. err=TypeError("Adagrad.__init__() got an unexpected keyword argument 'differentiable'"), type(err)=<class 'TypeError'>
    Returned to Spot: Validation loss: nan

    config: {'l1': 512, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.5050500035285950 | Loss: 1.3639718857765197 | Acc: 0.5050500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.5328500270843506 | Loss: 1.3052099388599396 | Acc: 0.5328500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.5652499794960022 | Loss: 1.2344603229284286 | Acc: 0.5652500000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5896000266075134 | Loss: 1.1868535377025604 | Acc: 0.5896000000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.5887500047683716 | Loss: 1.2098212986707688 | Acc: 0.5887500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5702499747276306 | Loss: 1.3166809049129486 | Acc: 0.5702500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5974500179290771 | Loss: 1.2765553975105286 | Acc: 0.5974500000000000.
    Early stopping at epoch 6
    Returned to Spot: Validation loss: 1.2765553975105286

    spotPython tuning: 1.1785970301628113 [##--------] 17.36%

    config: {'l1': 512, 'l2': 512, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.4888499975204468 | Loss: 1.3960691669464111 | Acc: 0.4888500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.5519499778747559 | Loss: 1.2605930910110474 | Acc: 0.5519500000000001.
    Epoch: 3 |

    MulticlassAccuracy: 0.5625500082969666 | Loss: 1.2682327481269837 | Acc: 0.5625500000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5820000171661377 | Loss: 1.2330140625953674 | Acc: 0.5820000000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.6009500026702881 | Loss: 1.1801210371971131 | Acc: 0.6009500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5918499827384949 | Loss: 1.3010232583999635 | Acc: 0.5918500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5907499790191650 | Loss: 1.4316253225326538 | Acc: 0.5907500000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.5793499946594238 | Loss: 1.5771554583549499 | Acc: 0.5793500000000000.
    Early stopping at epoch 7
    Returned to Spot: Validation loss: 1.5771554583549499
    spotPython tuning: 1.1785970301628113 [###-------] 34.04%

    config: {'l1': 64, 'l2': 128, 'lr_mult': 1.0, 'batch_size': 2, 'epochs': 8, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.4648500084877014 | Loss: 1.5890214798149769 | Acc: 0.4648500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.5056499838829041 | Loss: 1.4554801068477441 | Acc: 0.5056500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.5152999758720398 | Loss: 1.4293135994918207 | Acc: 0.5153000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5337499976158142 | Loss: 1.5508460156368367 | Acc: 0.5337499999999999.
    Epoch: 5 |

    MulticlassAccuracy: 0.5503500103950500 | Loss: 1.4691318791180024 | Acc: 0.5503500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5546500086784363 | Loss: 1.4323577481051282 | Acc: 0.5546500000000000.
    Early stopping at epoch 5
    Returned to Spot: Validation loss: 1.4323577481051282
    spotPython tuning: 1.1785970301628113 [######----] 55.97%

    config: {'l1': 64, 'l2': 256, 'lr_mult': 1.0, 'batch_size': 16, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.4751999974250793 | Loss: 1.4380010580539704 | Acc: 0.4752000000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.5193499922752380 | Loss: 1.3463098950862884 | Acc: 0.5193500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.5264999866485596 | Loss: 1.3246383390426635 | Acc: 0.5265000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5678499937057495 | Loss: 1.2190419143438340 | Acc: 0.5678500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.5468999743461609 | Loss: 1.2767102817535401 | Acc: 0.5469000000000001.
    Epoch: 6 |

    MulticlassAccuracy: 0.5749999880790710 | Loss: 1.2085392861127853 | Acc: 0.5750000000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5769000053405762 | Loss: 1.2117780505180360 | Acc: 0.5769000000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.5845500230789185 | Loss: 1.2138463154315948 | Acc: 0.5845500000000000.
    Epoch: 9 |

    MulticlassAccuracy: 0.5821999907493591 | Loss: 1.2590672193288803 | Acc: 0.5822000000000001.
    Early stopping at epoch 8
    Returned to Spot: Validation loss: 1.2590672193288803
    spotPython tuning: 1.1785970301628113 [########--] 75.36%

    config: {'l1': 128, 'l2': 16, 'lr_mult': 1.0, 'batch_size': 32, 'epochs': 16, 'k_folds': 0, 'patience': 3, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.4110499918460846 | Loss: 1.6076623956680298 | Acc: 0.4110500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.4602999985218048 | Loss: 1.4779073149681092 | Acc: 0.4603000000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.4859000146389008 | Loss: 1.4199016831398010 | Acc: 0.4859000000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.5196499824523926 | Loss: 1.3301249086380005 | Acc: 0.5196499999999999.
    Epoch: 5 |

    MulticlassAccuracy: 0.5414999723434448 | Loss: 1.2763008275985719 | Acc: 0.5415000000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5590000152587891 | Loss: 1.2436287584304810 | Acc: 0.5590000000000001.
    Epoch: 7 |

    MulticlassAccuracy: 0.5619500279426575 | Loss: 1.2420474333763123 | Acc: 0.5619499999999999.
    Epoch: 8 |

    MulticlassAccuracy: 0.5763999819755554 | Loss: 1.2015644022941590 | Acc: 0.5764000000000000.
    Epoch: 9 |

    MulticlassAccuracy: 0.5801500082015991 | Loss: 1.1826168160438537 | Acc: 0.5801500000000001.
    Epoch: 10 |

    MulticlassAccuracy: 0.5890499949455261 | Loss: 1.1751258714675903 | Acc: 0.5890500000000000.
    Epoch: 11 |

    MulticlassAccuracy: 0.5797500014305115 | Loss: 1.1968159104347229 | Acc: 0.5797500000000000.
    Epoch: 12 |

    MulticlassAccuracy: 0.5964000225067139 | Loss: 1.1581353195190429 | Acc: 0.5964000000000000.
    Epoch: 13 |

    MulticlassAccuracy: 0.5958999991416931 | Loss: 1.1588751288414001 | Acc: 0.5959000000000000.
    Epoch: 14 |

    MulticlassAccuracy: 0.5921000242233276 | Loss: 1.1875594395637512 | Acc: 0.5921000000000000.
    Epoch: 15 |

    MulticlassAccuracy: 0.5878000259399414 | Loss: 1.2220218329429626 | Acc: 0.5878000000000000.
    Early stopping at epoch 14
    Returned to Spot: Validation loss: 1.2220218329429626
    spotPython tuning: 1.1785970301628113 [##########] 100.00% Done...

    <spotPython.spot.spot.Spot at 0x296de1630>

## 12.9 Step 9: Tensorboard

The textual output shown in the console (or code cell) can be visualized with Tensorboard.

### 12.9.1 Tensorboard: Start Tensorboard

Start TensorBoard through the command line to visualize data you logged. Specify the root log directory as used in `fun_control = fun_control_init(task="regression", tensorboard_path="runs/24_spot_torch_regression")` as the `tensorboard_path`. The argument logdir points to directory where TensorBoard will look to find event files that it can display. TensorBoard will recursively walk the directory structure rooted at logdir, looking for .tfevents. files.

    tensorboard --logdir=runs

Go to the URL it provides or to http://localhost:6006/. The following figures show some screenshots of Tensorboard.

Figure 12.1: Tensorboard

Figure 12.2: Tensorboard

### 12.9.2 Saving the State of the Notebook

The state of the notebook can be saved and reloaded as follows:

    import pickle
    SAVE = False
    LOAD = False

    if SAVE:
        result_file_name = "res_" + experiment_name + ".pkl"
        with open(result_file_name, 'wb') as f:
            pickle.dump(spot_tuner, f)

    if LOAD:
        result_file_name = "add_the_name_of_the_result_file_here.pkl"
        with open(result_file_name, 'rb') as f:
            spot_tuner =  pickle.load(f)

## 12.10 Step 10: Results

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.

    spot_tuner.plot_progress(log_y=False,
        filename="./figures/" + experiment_name+"_progress.png")

Progress plot. Black dots denote results from the initial design. Red dots illustrate the improvement found by the surrogate model based optimization.

?fig-progress shows a typical behaviour that can be observed in many hyperparameter studies (Bartz et al. 2022): the largest improvement is obtained during the evaluation of the initial design. The surrogate model based optimization-optimization with the surrogate refines the results. ?fig-progress also illustrates one major difference between `ray[tune]` as used in PyTorch (2023a) and `spotPython`: the `ray[tune]` uses a random search and will generate results similar to the black dots, whereas `spotPython` uses a surrogate model based optimization and presents results represented by red dots in ?fig-progress. The surrogate model based optimization is considered to be more efficient than a random search, because the surrogate model guides the search towards promising regions in the hyperparameter space.

In addition to the improved (“optimized”) hyperparameter values, `spotPython` allows a statistical analysis, e.g., a sensitivity analysis, of the results. We can print the results of the hyperparameter tuning, see ?tbl-results. The table shows the hyperparameters, their types, default values, lower and upper bounds, and the transformation function. The column “tuned” shows the tuned values. The column “importance” shows the importance of the hyperparameters. The column “stars” shows the importance of the hyperparameters in stars. The importance is computed by the SPOT software.

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control=fun_control, spot=spot_tuner))

    | name         | type   | default   |   lower |   upper |   tuned | transform             |   importance | stars   |
    |--------------|--------|-----------|---------|---------|---------|-----------------------|--------------|---------|
    | l1           | int    | 5         |     2.0 |     9.0 |     7.0 | transform_power_2_int |         0.00 |         |
    | l2           | int    | 5         |     2.0 |     9.0 |     3.0 | transform_power_2_int |         0.00 |         |
    | lr_mult      | float  | 1.0       |     1.0 |     1.0 |     1.0 | None                  |         0.00 |         |
    | batch_size   | int    | 4         |     1.0 |     5.0 |     5.0 | transform_power_2_int |         1.84 | *       |
    | epochs       | int    | 3         |     3.0 |     4.0 |     4.0 | transform_power_2_int |       100.00 | ***     |
    | k_folds      | int    | 1         |     0.0 |     0.0 |     0.0 | None                  |         0.00 |         |
    | patience     | int    | 5         |     3.0 |     3.0 |     3.0 | None                  |         0.00 |         |
    | optimizer    | factor | SGD       |     0.0 |     9.0 |     3.0 | None                  |         0.01 |         |
    | sgd_momentum | float  | 0.0       |     0.9 |     0.9 |     0.9 | None                  |         0.00 |         |

To visualize the most important hyperparameters, `spotPython` provides the function `plot_importance`. The following code generates the importance plot from ?fig-importance.

    spot_tuner.plot_importance(threshold=0.025,
        filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

### 12.10.1 Get the Tuned Architecture (SPOT Results)

The architecture of the `spotPython` model can be obtained as follows. First, the numerical representation of the hyperparameters are obtained, i.e., the numpy array `X` is generated. This array is then used to generate the model `model_spot` by the function `get_one_core_model_from_X`. The model `model_spot` has the following architecture:

    from spotPython.hyperparameters.values import get_one_core_model_from_X
    X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
    model_spot = get_one_core_model_from_X(X, fun_control)
    model_spot

    Net_CIFAR10(
      (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
      (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
      (fc1): Linear(in_features=400, out_features=128, bias=True)
      (fc2): Linear(in_features=128, out_features=8, bias=True)
      (fc3): Linear(in_features=8, out_features=10, bias=True)
    )

### 12.10.2 Get Default Hyperparameters

In a similar manner as in Section 12.10.1, the default hyperparameters can be obtained.

    # fun_control was modified, we generate a new one with the original
    # default hyperparameters
    from spotPython.hyperparameters.values import get_one_core_model_from_X
    fc = fun_control
    fc.update({"core_model_hyper_dict":
        hyper_dict[fun_control["core_model"].__name__]})
    model_default = get_one_core_model_from_X(X_start, fun_control=fc)
    model_default

    Net_CIFAR10(
      (conv1): Conv2d(3, 6, kernel_size=(5, 5), stride=(1, 1))
      (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))
      (fc1): Linear(in_features=400, out_features=32, bias=True)
      (fc2): Linear(in_features=32, out_features=32, bias=True)
      (fc3): Linear(in_features=32, out_features=10, bias=True)
    )

### 12.10.3 Evaluation of the Default Architecture

The method `train_tuned` takes a model architecture without trained weights and trains this model with the train data. The train data is split into train and validation data. The validation data is used for early stopping. The trained model weights are saved as a dictionary.

This evaluation is similar to the final evaluation in PyTorch (2023a).

    from spotPython.torch.traintest import (
        train_tuned,
        test_tuned,
        )
    train_tuned(net=model_default, train_dataset=train, shuffle=True,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            device = fun_control["device"], show_batch_interval=1_000_000,
            path=None,
            task=fun_control["task"],)

    test_tuned(net=model_default, test_dataset=test,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            shuffle=False,
            device = fun_control["device"],
            task=fun_control["task"],)

    Epoch: 1 |

    MulticlassAccuracy: 0.1020999997854233 | Loss: 2.3032254779815675 | Acc: 0.1021000000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.1037999987602234 | Loss: 2.3003110511779785 | Acc: 0.1038000000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.1328500062227249 | Loss: 2.2962776536941529 | Acc: 0.1328500000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.1394499987363815 | Loss: 2.2892701484680176 | Acc: 0.1394500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.1676999926567078 | Loss: 2.2727036296844481 | Acc: 0.1677000000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.1837500035762787 | Loss: 2.2302733320236205 | Acc: 0.1837500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.1986999958753586 | Loss: 2.1764420920372007 | Acc: 0.1987000000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.2235500067472458 | Loss: 2.1471230669021608 | Acc: 0.2235500000000000.
    Returned to Spot: Validation loss: 2.147123066902161

    MulticlassAccuracy: 0.2236000001430511 | Loss: 2.1441712369918822 | Acc: 0.2236000000000000.
    Final evaluation: Validation loss: 2.144171236991882
    Final evaluation: Validation metric: 0.22360000014305115
    ----------------------------------------------

    (2.144171236991882, nan, tensor(0.2236))

### 12.10.4 Evaluation of the Tuned Architecture

The following code trains the model `model_spot`.

If `path` is set to a filename, e.g., `path = "model_spot_trained.pt"`, the weights of the trained model will be saved to this file.

If `path` is set to a filename, e.g., `path = "model_spot_trained.pt"`, the weights of the trained model will be loaded from this file.

    train_tuned(net=model_spot, train_dataset=train,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            shuffle=True,
            device = fun_control["device"],
            path=None,
            task=fun_control["task"],)
    test_tuned(net=model_spot, test_dataset=test,
                shuffle=False,
                loss_function=fun_control["loss_function"],
                metric=fun_control["metric_torch"],
                device = fun_control["device"],
                task=fun_control["task"],)

    Epoch: 1 |

    MulticlassAccuracy: 0.3473500013351440 | Loss: 1.7150537954330445 | Acc: 0.3473500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.3957499861717224 | Loss: 1.6454636848449706 | Acc: 0.3957500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.4472500085830688 | Loss: 1.5022996424674988 | Acc: 0.4472500000000000.
    Epoch: 4 |

    MulticlassAccuracy: 0.4896500110626221 | Loss: 1.4034859374046327 | Acc: 0.4896500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.4974499940872192 | Loss: 1.4032018181800843 | Acc: 0.4974500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5215499997138977 | Loss: 1.3365078643798829 | Acc: 0.5215500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.5320000052452087 | Loss: 1.3101746562004088 | Acc: 0.5320000000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.5446500182151794 | Loss: 1.2812147528648377 | Acc: 0.5446500000000000.
    Epoch: 9 |

    MulticlassAccuracy: 0.5465999841690063 | Loss: 1.2808432787895203 | Acc: 0.5466000000000000.
    Epoch: 10 |

    MulticlassAccuracy: 0.5641000270843506 | Loss: 1.2373182715415953 | Acc: 0.5641000000000000.
    Epoch: 11 |

    MulticlassAccuracy: 0.5662500262260437 | Loss: 1.2444315943717956 | Acc: 0.5662500000000000.
    Epoch: 12 |

    MulticlassAccuracy: 0.5701000094413757 | Loss: 1.2425961187362671 | Acc: 0.5701000000000001.
    Epoch: 13 |

    MulticlassAccuracy: 0.5824000239372253 | Loss: 1.2197082972526549 | Acc: 0.5824000000000000.
    Epoch: 14 |

    MulticlassAccuracy: 0.5792999863624573 | Loss: 1.2425022669792176 | Acc: 0.5793000000000000.
    Epoch: 15 |

    MulticlassAccuracy: 0.5849000215530396 | Loss: 1.2452355551719665 | Acc: 0.5849000000000000.
    Epoch: 16 |

    MulticlassAccuracy: 0.5925999879837036 | Loss: 1.2358321579933167 | Acc: 0.5926000000000000.
    Early stopping at epoch 15
    Returned to Spot: Validation loss: 1.2358321579933167

    MulticlassAccuracy: 0.5950000286102295 | Loss: 1.2175832162269007 | Acc: 0.5950000000000000.
    Final evaluation: Validation loss: 1.2175832162269007
    Final evaluation: Validation metric: 0.5950000286102295
    ----------------------------------------------

    (1.2175832162269007, nan, tensor(0.5950))

### 12.10.5 Detailed Hyperparameter Plots

The contour plots in this section visualize the interactions of the three most important hyperparameters. Since some of these hyperparameters take fatorial or integer values, sometimes step-like fitness landcapes (or response surfaces) are generated. SPOT draws the interactions of the main hyperparameters by default. It is also possible to visualize all interactions.

    filename = "./figures/" + experiment_name
    spot_tuner.plot_important_hyperparameter_contour(filename=filename)

    batch_size:  1.8353888605258115
    epochs:  100.0

Contour plots.

The figures (?fig-contour) show the contour plots of the loss as a function of the hyperparameters. These plots are very helpful for benchmark studies and for understanding neural networks. `spotPython` provides additional tools for a visual inspection of the results and give valuable insights into the hyperparameter tuning process. This is especially useful for model explainability, transparency, and trustworthiness. In addition to the contour plots, ?fig-parallel shows the parallel plot of the hyperparameters.

    spot_tuner.parallel_plot()

Parallel coordinates plots

## 12.11 Summary and Outlook

This tutorial presents the hyperparameter tuning open source software `spotPython` for `PyTorch`. To show its basic features, a comparison with the “official” `PyTorch` hyperparameter tuning tutorial (PyTorch 2023a) is presented. Some of the advantages of `spotPython` are:

- Numerical and categorical hyperparameters.
- Powerful surrogate models.
- Flexible approach and easy to use.
- Simple JSON files for the specification of the hyperparameters.
- Extension of default and user specified network classes.
- Noise handling techniques.
- Interaction with `tensorboard`.

Currently, only rudimentary parallel and distributed neural network training is possible, but these capabilities will be extended in the future. The next version of `spotPython` will also include a more detailed documentation and more examples.

Important

Important: This tutorial does not present a complete benchmarking study (Bartz-Beielstein et al. 2020). The results are only preliminary and highly dependent on the local configuration (hard- and software). Our goal is to provide a first impression of the performance of the hyperparameter tuning package `spotPython`. To demonstrate its capabilities, a quick comparison with `ray[tune]` was performed. `ray[tune]` was chosen, because it is presented as “an industry standard tool for distributed hyperparameter tuning.” The results should be interpreted with care.

## 12.12 Appendix

### 12.12.1 Sample Output From Ray Tune’s Run

The output from `ray[tune]` could look like this (PyTorch 2023b):

    Number of trials: 10 (10 TERMINATED)
    ------+------+-------------+--------------+---------+------------+--------------------+
    |   l1 |   l2 |          lr |   batch_size |    loss |   accuracy | training_iteration |
    +------+------+-------------+--------------+---------+------------+--------------------|
    |   64 |    4 | 0.00011629  |            2 | 1.87273 |     0.244  |                  2 |
    |   32 |   64 | 0.000339763 |            8 | 1.23603 |     0.567  |                  8 |
    |    8 |   16 | 0.00276249  |           16 | 1.1815  |     0.5836 |                 10 |
    |    4 |   64 | 0.000648721 |            4 | 1.31131 |     0.5224 |                  8 |
    |   32 |   16 | 0.000340753 |            8 | 1.26454 |     0.5444 |                  8 |
    |    8 |    4 | 0.000699775 |            8 | 1.99594 |     0.1983 |                  2 |
    |  256 |    8 | 0.0839654   |           16 | 2.3119  |     0.0993 |                  1 |
    |   16 |  128 | 0.0758154   |           16 | 2.33575 |     0.1327 |                  1 |
    |   16 |    8 | 0.0763312   |           16 | 2.31129 |     0.1042 |                  4 |
    |  128 |   16 | 0.000124903 |            4 | 2.26917 |     0.1945 |                  1 |
    +-----+------+------+-------------+--------------+---------+------------+--------------------+
    Best trial config: {'l1': 8, 'l2': 16, 'lr': 0.00276249, 'batch_size': 16, 'data_dir': '...'}
    Best trial final validation loss: 1.181501
    Best trial final validation accuracy: 0.5836
    Best trial test set accuracy: 0.5806

Bartz, Eva, Thomas Bartz-Beielstein, Martin Zaefferer, and Olaf Mersmann, eds. 2022. Hyperparameter Tuning for Machine and Deep Learning with R - A Practical Guide. Springer.
Bartz-Beielstein, Thomas, Carola Doerr, Jakob Bossek, Sowmya Chandrasekaran, Tome Eftimov, Andreas Fischbach, Pascal Kerschke, et al. 2020. “Benchmarking in Optimization: Best Practice and Open Issues.” arXiv. https://arxiv.org/abs/2007.03488.
Montiel, Jacob, Max Halford, Saulo Martiello Mastelini, Geoffrey Bolmier, Raphael Sourty, Robin Vaysse, Adil Zouitine, et al. 2021. “River: Machine Learning for Streaming Data in Python.”
PyTorch. 2023a. “Hyperparameter Tuning with Ray Tune.” https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html.
———. 2023b. “Training a Classifier.” https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html.

1. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.↩︎

2. We were not able to install `Ray Tune` on our system. Therefore, we used the results from the `PyTorch` tutorial.↩︎