# 11 HPT: PyTorch With fashionMNIST

In this tutorial, we will show how `spotPython` can be integrated into the `PyTorch` training workflow.

This document refers to the following software versions:

- `python`: 3.10.10
- `torch`: 2.0.1
- `torchvision`: 0.15.0

    pip list | grep  "spot[RiverPython]"

    spotPython                 0.2.38
    spotRiver                  0.0.93

    Note: you may need to restart the kernel to use updated packages.

`spotPython` can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.

    !pip install spotPython

- Uncomment the following lines if you want to for (re-)installation the latest version of `spotPython` from gitHub.

    # import sys
    # !{sys.executable} -m pip install --upgrade build
    # !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

## 11.1 Step 1: Setup

Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size and the device that is used.

Caution: Run time and initial design size should be increased for real experiments

- MAX_TIME is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
- INIT_SIZE is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.

Note: Device selection

- The device can be selected by setting the variable `DEVICE`.
- Since we are using a simple neural net, the setting `"cpu"` is preferred (on Mac).
- If you have a GPU, you can use `"cuda:0"` instead.
- If DEVICE is set to `None`, `spotPython` will automatically select the device.
  - This might result in `"mps"` on Macs, which is not the best choice for simple neural nets.

    MAX_TIME = 1
    INIT_SIZE = 5
    DEVICE = "cpu" # "cuda:0"

    from spotPython.utils.device import getDevice
    DEVICE = getDevice(DEVICE)
    print(DEVICE)

    cpu

    import os
    import copy
    import socket
    from datetime import datetime
    from dateutil.tz import tzlocal
    start_time = datetime.now(tzlocal())
    HOSTNAME = socket.gethostname().split(".")[0]
    experiment_name = '11-torch' + "_" + HOSTNAME + "_" + str(MAX_TIME) + "min_" + str(INIT_SIZE) + "init_" + str(start_time).split(".", 1)[0].replace(' ', '_')
    experiment_name = experiment_name.replace(':', '-')
    print(experiment_name)
    if not os.path.exists('./figures'):
        os.makedirs('./figures')

    11-torch_bartz09_1min_5init_2023-06-19_02-28-05

## 11.2 Step 2: Initialization of the Empty `fun_control` Dictionary

`spotPython` uses a Python dictionary for storing the information required for the hyperparameter tuning process, which was described in Section 14.2.

Caution: Tensorboard does not work under Windows

- Since tensorboard does not work under Windows, we recommend setting the parameter `tensorboard_path` to `None` if you are working under Windows.

    from spotPython.utils.init import fun_control_init
    fun_control = fun_control_init(task="classification",
        tensorboard_path="runs/11_spot_hpt_torch_fashion_mnist",
        device=DEVICE)

## 11.3 Step 3: PyTorch Data Loading

### 11.3.1 Load fashionMNIST Data

    from torchvision import datasets, transforms
    from torchvision.transforms import ToTensor
    def load_data(data_dir="./data"):
        # Download training data from open datasets.
        training_data = datasets.FashionMNIST(
            root=data_dir,
            train=True,
            download=True,
            transform=ToTensor(),
        )
        # Download test data from open datasets.
        test_data = datasets.FashionMNIST(
            root=data_dir,
            train=False,
            download=True,
            transform=ToTensor(),
        )
        return training_data, test_data

    train, test = load_data()
    train.data.shape, test.data.shape

    (torch.Size([60000, 28, 28]), torch.Size([10000, 28, 28]))

    n_samples = len(train)
    # add the dataset to the fun_control
    fun_control.update({"data": None,
                   "train": train,
                   "test": test,
                   "n_samples": n_samples,
                   "target_column": None})

## 11.4 Step 4: Specification of the Preprocessing Model

After the training and test data are specified and added to the `fun_control` dictionary, `spotPython` allows the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables, see Section 14.4. This feature is not used here, so we do not change the default value (which is `None`).

## 11.5 Step 5: Select Model (`algorithm`) and `core_model_hyper_dict`

`spotPython` implements a class which is similar to the class described in the PyTorch tutorial. The class is called `Net_fashionMNIST` and is implemented in the file `netfashionMNIST.py`. The class is imported here.

    from torch import nn
    import spotPython.torch.netcore as netcore

    class Net_fashionMNIST(netcore.Net_Core):
        def __init__(self, l1, l2, lr_mult, batch_size, epochs, k_folds, patience, optimizer, sgd_momentum):
            super(Net_fashionMNIST, self).__init__(
                lr_mult=lr_mult,
                batch_size=batch_size,
                epochs=epochs,
                k_folds=k_folds,
                patience=patience,
                optimizer=optimizer,
                sgd_momentum=sgd_momentum,
            )
            self.flatten = nn.Flatten()
            self.linear_relu_stack = nn.Sequential(
                nn.Linear(28 * 28, l1),
                nn.ReLU(),
                nn.Linear(l1, l2),
                nn.ReLU(),
                nn.Linear(l2, 10)
            )

        def forward(self, x):
            x = self.flatten(x)
            logits = self.linear_relu_stack(x)
            return logits

This class inherits from the class `Net_Core` which is implemented in the file `netcore.py`, see Section 14.5.1.

    from spotPython.data.torch_hyper_dict import TorchHyperDict
    from spotPython.torch.netfashionMNIST import Net_fashionMNIST
    from spotPython.hyperparameters.values import add_core_model_to_fun_control
    fun_control = add_core_model_to_fun_control(core_model=Net_fashionMNIST,
                                  fun_control=fun_control,
                                  hyper_dict=TorchHyperDict,
                                  filename=None)

### 11.5.1 The Search Space

### 11.5.2 Configuring the Search Space With spotPython

#### 11.5.2.1 The `hyper_dict` Hyperparameters for the Selected Algorithm

`spotPython` uses `JSON` files for the specification of the hyperparameters, which were described in Section 14.5.5.

The corresponding entries for the `core_model` class are shown below.

    fun_control['core_model_hyper_dict']

    {'l1': {'type': 'int',
      'default': 5,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 9},
     'l2': {'type': 'int',
      'default': 5,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 9},
     'lr_mult': {'type': 'float',
      'default': 1.0,
      'transform': 'None',
      'lower': 0.1,
      'upper': 10.0},
     'batch_size': {'type': 'int',
      'default': 4,
      'transform': 'transform_power_2_int',
      'lower': 1,
      'upper': 4},
     'epochs': {'type': 'int',
      'default': 3,
      'transform': 'transform_power_2_int',
      'lower': 3,
      'upper': 4},
     'k_folds': {'type': 'int',
      'default': 1,
      'transform': 'None',
      'lower': 1,
      'upper': 1},
     'patience': {'type': 'int',
      'default': 5,
      'transform': 'None',
      'lower': 2,
      'upper': 10},
     'optimizer': {'levels': ['Adadelta',
       'Adagrad',
       'Adam',
       'AdamW',
       'SparseAdam',
       'Adamax',
       'ASGD',
       'NAdam',
       'RAdam',
       'RMSprop',
       'Rprop',
       'SGD'],
      'type': 'factor',
      'default': 'SGD',
      'transform': 'None',
      'core_model_parameter_type': 'str',
      'lower': 0,
      'upper': 12},
     'sgd_momentum': {'type': 'float',
      'default': 0.0,
      'transform': 'None',
      'lower': 0.0,
      'upper': 1.0}}

## 11.6 Step 6: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

`spotPython` provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 14.6.

### 11.6.1 Modify hyperparameter of type numeric and integer (boolean)

The hyperparameter `k_folds` is not used, it is de-activated here by setting the lower and upper bound to the same value.

Caution: Small net size, number of epochs, and patience for demonstration purposes

- Net sizes `l1` and `l2` as well as `epochs` and `patience` are set to small values for demonstration purposes. These values are too small for a real application.
- More resonable values are, e.g.:
  - `fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[2, 7])`
  - `fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[7, 9])` and
  - `fun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 7])`

    from spotPython.hyperparameters.values import modify_hyper_parameter_bounds
    fun_control = modify_hyper_parameter_bounds(fun_control, "k_folds", bounds=[0, 0])
    fun_control = modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 2])
    fun_control = modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[2, 3])
    fun_control = modify_hyper_parameter_bounds(fun_control, "l1", bounds=[2, 5])
    fun_control = modify_hyper_parameter_bounds(fun_control, "l2", bounds=[2, 5])

### 11.6.2 Modify hyperparameter of type factor

    from spotPython.hyperparameters.values import modify_hyper_parameter_levels
    fun_control = modify_hyper_parameter_levels(fun_control, "optimizer",["Adam", "AdamW", "Adamax", "NAdam"])

### 11.6.3 Optimizers

Optimizers are described in Section 14.6.1.

    fun_control = modify_hyper_parameter_bounds(fun_control,
        "lr_mult", bounds=[1e-3, 1e-3])
    fun_control = modify_hyper_parameter_bounds(fun_control,
        "sgd_momentum", bounds=[0.9, 0.9])

## 11.7 Step 7: Selection of the Objective (Loss) Function

### 11.7.1 Evaluation

The evaluation procedure requires the specification of two elements:

1. the way how the data is split into a train and a test set and
2. the loss function (and a metric).

These are described in Section 19.7.1.

The key `"loss_function"` specifies the loss function which is used during the optimization, see Section 14.7.5.

We will use CrossEntropy loss for the multiclass-classification task.

    from torch.nn import CrossEntropyLoss
    loss_function = CrossEntropyLoss()
    fun_control.update({
            "loss_function": loss_function,
            "shuffle": True,
            "eval":  "train_hold_out"
            })

### 11.7.2 Metric

    from torchmetrics import Accuracy
    metric_torch = Accuracy(task="multiclass", num_classes=10).to(fun_control["device"])
    fun_control.update({"metric_torch": metric_torch})

## 11.8 Step 8: Calling the SPOT Function

### 11.8.1 Preparing the SPOT Call

The following code passes the information about the parameter ranges and bounds to `spot`.

    # extract the variable types, names, and bounds
    from spotPython.hyperparameters.values import (get_bound_values,
        get_var_name,
        get_var_type,)
    var_type = get_var_type(fun_control)
    var_name = get_var_name(fun_control)
    fun_control.update({"var_type": var_type,
                        "var_name": var_name})
    lower = get_bound_values(fun_control, "lower")
    upper = get_bound_values(fun_control, "upper")

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control))

    | name         | type   | default   |   lower |   upper | transform             |
    |--------------|--------|-----------|---------|---------|-----------------------|
    | l1           | int    | 5         |   2     |   5     | transform_power_2_int |
    | l2           | int    | 5         |   2     |   5     | transform_power_2_int |
    | lr_mult      | float  | 1.0       |   0.001 |   0.001 | None                  |
    | batch_size   | int    | 4         |   1     |   4     | transform_power_2_int |
    | epochs       | int    | 3         |   2     |   3     | transform_power_2_int |
    | k_folds      | int    | 1         |   0     |   0     | None                  |
    | patience     | int    | 5         |   2     |   2     | None                  |
    | optimizer    | factor | SGD       |   0     |   3     | None                  |
    | sgd_momentum | float  | 0.0       |   0.9   |   0.9   | None                  |

### 11.8.2 The Objective Function `fun_torch`

The objective function `fun_torch` is selected next. It implements an interface from `PyTorch`’s training, validation, and testing methods to `spotPython`.

    from spotPython.fun.hypertorch import HyperTorch
    fun = HyperTorch().fun_torch

### 11.8.3 Starting the Hyperparameter Tuning

    import numpy as np
    from spotPython.spot import spot
    from math import inf
    spot_tuner = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = inf,
                       fun_repeats = 1,
                       max_time = MAX_TIME,
                       noise = False,
                       tolerance_x = np.sqrt(np.spacing(1)),
                       var_type = var_type,
                       var_name = var_name,
                       infill_criterion = "y",
                       n_points = 1,
                       seed=123,
                       log_level = 50,
                       show_models= False,
                       show_progress= True,
                       fun_control = fun_control,
                       design_control={"init_size": INIT_SIZE,
                                       "repeats": 1},
                       surrogate_control={"noise": True,
                                          "cod_type": "norm",
                                          "min_theta": -4,
                                          "max_theta": 3,
                                          "n_theta": len(var_name),
                                          "model_fun_evals": 10_000,
                                          "log_level": 50
                                          })
    spot_tuner.run(X_start=X_start)

    config: {'l1': 16, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 16, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.0997499972581863 | Loss: 2.3140821170806887 | Acc: 0.0997500000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.1206666678190231 | Loss: 2.2892945305506389 | Acc: 0.1206666666666667.
    Epoch: 3 |

    MulticlassAccuracy: 0.1177083328366280 | Loss: 2.2667990508079527 | Acc: 0.1177083333333333.
    Epoch: 4 |

    MulticlassAccuracy: 0.1209166646003723 | Loss: 2.2469974958101910 | Acc: 0.1209166666666667.
    Epoch: 5 |

    MulticlassAccuracy: 0.1688333302736282 | Loss: 2.2289535454114278 | Acc: 0.1688333333333333.
    Epoch: 6 |

    MulticlassAccuracy: 0.1907500028610229 | Loss: 2.2126366780598956 | Acc: 0.1907500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.1978749930858612 | Loss: 2.1974748126665751 | Acc: 0.1978750000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.2006250023841858 | Loss: 2.1834416908423107 | Acc: 0.2006250000000000.
    Returned to Spot: Validation loss: 2.1834416908423107

    config: {'l1': 8, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'Adamax', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.0989999994635582 | Loss: 2.3134303503036500 | Acc: 0.0990000000000000.
    Epoch: 2 |

    MulticlassAccuracy: 0.0994166657328606 | Loss: 2.3042378049691519 | Acc: 0.0994166666666667.
    Epoch: 3 |

    MulticlassAccuracy: 0.0999583303928375 | Loss: 2.2962032955487568 | Acc: 0.0999583333333333.
    Epoch: 4 |

    MulticlassAccuracy: 0.1011250019073486 | Loss: 2.2879681829611460 | Acc: 0.1011250000000000.
    Returned to Spot: Validation loss: 2.287968182961146

    config: {'l1': 32, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.2570833265781403 | Loss: 2.1080865803857645 | Acc: 0.2570833333333333.
    Epoch: 2 |

    MulticlassAccuracy: 0.3926250040531158 | Loss: 1.8708737857242426 | Acc: 0.3926250000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.3989166617393494 | Loss: 1.6699255141715208 | Acc: 0.3989166666666666.
    Epoch: 4 |

    MulticlassAccuracy: 0.4268333315849304 | Loss: 1.5093953567594289 | Acc: 0.4268333333333333.
    Epoch: 5 |

    MulticlassAccuracy: 0.4982500076293945 | Loss: 1.3648724806085228 | Acc: 0.4982500000000000.
    Epoch: 6 |

    MulticlassAccuracy: 0.5611666440963745 | Loss: 1.2497935280787449 | Acc: 0.5611666666666667.
    Epoch: 7 |

    MulticlassAccuracy: 0.6253749728202820 | Loss: 1.1577660601995885 | Acc: 0.6253750000000000.
    Epoch: 8 |

    MulticlassAccuracy: 0.6429583430290222 | Loss: 1.0825004865694792 | Acc: 0.6429583333333333.
    Returned to Spot: Validation loss: 1.0825004865694792

    config: {'l1': 4, 'l2': 8, 'lr_mult': 0.001, 'batch_size': 4, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.0867083296179771 | Loss: 2.3195563216606776 | Acc: 0.0867083333333333.
    Epoch: 2 |

    MulticlassAccuracy: 0.1370416730642319 | Loss: 2.3075901949405671 | Acc: 0.1370416666666667.
    Epoch: 3 |

    MulticlassAccuracy: 0.1562916636466980 | Loss: 2.2953362877368928 | Acc: 0.1562916666666667.
    Epoch: 4 |

    MulticlassAccuracy: 0.1665833294391632 | Loss: 2.2812995012998583 | Acc: 0.1665833333333333.
    Returned to Spot: Validation loss: 2.2812995012998583

    config: {'l1': 16, 'l2': 32, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'Adam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.0997916683554649 | Loss: 2.2726654102802275 | Acc: 0.0997916666666667.
    Epoch: 2 |

    MulticlassAccuracy: 0.0997916683554649 | Loss: 2.2293555783828101 | Acc: 0.0997916666666667.
    Epoch: 3 |

    MulticlassAccuracy: 0.0997916683554649 | Loss: 2.1818058302005130 | Acc: 0.0997916666666667.
    Epoch: 4 |

    MulticlassAccuracy: 0.1436666697263718 | Loss: 2.1319178236722944 | Acc: 0.1436666666666667.
    Epoch: 5 |

    MulticlassAccuracy: 0.1882083266973495 | Loss: 2.0800701110760369 | Acc: 0.1882083333333333.
    Epoch: 6 |

    MulticlassAccuracy: 0.2377499938011169 | Loss: 2.0257184394200642 | Acc: 0.2377500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.2879583239555359 | Loss: 1.9705022779305776 | Acc: 0.2879583333333333.
    Epoch: 8 |

    MulticlassAccuracy: 0.3180416524410248 | Loss: 1.9157442781527838 | Acc: 0.3180416666666667.
    Returned to Spot: Validation loss: 1.9157442781527838

    config: {'l1': 8, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 8, 'epochs': 8, 'k_folds': 0, 'patience': 2, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.1180833354592323 | Loss: 2.2814749220212303 | Acc: 0.1180833333333333.
    Epoch: 2 |

    MulticlassAccuracy: 0.1931666731834412 | Loss: 2.2344061804215114 | Acc: 0.1931666666666667.
    Epoch: 3 |

    MulticlassAccuracy: 0.1977083384990692 | Loss: 2.1826921091874438 | Acc: 0.1977083333333333.
    Epoch: 4 |

    MulticlassAccuracy: 0.1992499977350235 | Loss: 2.1234583709637325 | Acc: 0.1992500000000000.
    Epoch: 5 |

    MulticlassAccuracy: 0.2000833302736282 | Loss: 2.0587348871231077 | Acc: 0.2000833333333333.
    Epoch: 6 |

    MulticlassAccuracy: 0.1993750035762787 | Loss: 1.9904835938612619 | Acc: 0.1993750000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.1974583268165588 | Loss: 1.9215697557131450 | Acc: 0.1974583333333333.
    Epoch: 8 |

    MulticlassAccuracy: 0.2007499933242798 | Loss: 1.8549580786625544 | Acc: 0.2007500000000000.
    Returned to Spot: Validation loss: 1.8549580786625544
    spotPython tuning: 1.0825004865694792 [####------] 43.42%

    config: {'l1': 16, 'l2': 16, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'AdamW', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.1014166697859764 | Loss: 2.2889870933294296 | Acc: 0.1014166666666667.
    Epoch: 2 |

    MulticlassAccuracy: 0.1514583379030228 | Loss: 2.2382834155261517 | Acc: 0.1514583333333333.
    Epoch: 3 |

    MulticlassAccuracy: 0.1802083402872086 | Loss: 2.1784320760667324 | Acc: 0.1802083333333333.
    Epoch: 4 |

    MulticlassAccuracy: 0.1871666610240936 | Loss: 2.1112670509417852 | Acc: 0.1871666666666667.
    Returned to Spot: Validation loss: 2.1112670509417852

    spotPython tuning: 1.0825004865694792 [##########] 98.70%

    config: {'l1': 32, 'l2': 32, 'lr_mult': 0.001, 'batch_size': 2, 'epochs': 4, 'k_folds': 0, 'patience': 2, 'optimizer': 'NAdam', 'sgd_momentum': 0.9}
    Epoch: 1 |

    MulticlassAccuracy: 0.2952916622161865 | Loss: 2.0284752748111883 | Acc: 0.2952916666666667.
    Epoch: 2 |

    MulticlassAccuracy: 0.4288333356380463 | Loss: 1.6900148228754599 | Acc: 0.4288333333333333.
    Epoch: 3 |

    MulticlassAccuracy: 0.5361666679382324 | Loss: 1.4265262978275617 | Acc: 0.5361666666666667.
    Epoch: 4 |

    MulticlassAccuracy: 0.5820833444595337 | Loss: 1.2350391030311585 | Acc: 0.5820833333333333.
    Returned to Spot: Validation loss: 1.2350391030311585
    spotPython tuning: 1.0825004865694792 [##########] 100.00% Done...

    <spotPython.spot.spot.Spot at 0x294ab2f80>

## 11.9 Step 9: Tensorboard

The textual output shown in the console (or code cell) can be visualized with Tensorboard as described in Section 14.9, see also the description in the documentation: Tensorboard.

## 11.10 Step 10: Results

After the hyperparameter tuning run is finished, the results can be analyzed as described in Section 14.10.

    SAVE = False
    LOAD = False

    if SAVE:
        result_file_name = "res_" + experiment_name + ".pkl"
        with open(result_file_name, 'wb') as f:
            pickle.dump(spot_tuner, f)

    if LOAD:
        result_file_name = "ADD THE NAME here, e.g.: res_ch10-friedman-hpt-0_maans03_60min_20init_1K_2023-04-14_10-11-19.pkl"
        with open(result_file_name, 'rb') as f:
            spot_tuner =  pickle.load(f)

After the hyperparameter tuning run is finished, the progress of the hyperparameter tuning can be visualized. The following code generates the progress plot from ?fig-progress.

    spot_tuner.plot_progress(log_y=False,
        filename="./figures/" + experiment_name+"_progress.png")

Progress plot. Black dots denote results from the initial design. Red dots illustrate the improvement found by the surrogate model based optimization.

- Print the results

    print(gen_design_table(fun_control=fun_control,
        spot=spot_tuner))

    | name         | type   | default   |   lower |   upper |   tuned | transform             |   importance | stars   |
    |--------------|--------|-----------|---------|---------|---------|-----------------------|--------------|---------|
    | l1           | int    | 5         |     2.0 |     5.0 |     5.0 | transform_power_2_int |       100.00 | ***     |
    | l2           | int    | 5         |     2.0 |     5.0 |     4.0 | transform_power_2_int |         0.01 |         |
    | lr_mult      | float  | 1.0       |   0.001 |   0.001 |   0.001 | None                  |         0.00 |         |
    | batch_size   | int    | 4         |     1.0 |     4.0 |     1.0 | transform_power_2_int |         0.01 |         |
    | epochs       | int    | 3         |     2.0 |     3.0 |     3.0 | transform_power_2_int |         0.01 |         |
    | k_folds      | int    | 1         |     0.0 |     0.0 |     0.0 | None                  |         0.00 |         |
    | patience     | int    | 5         |     2.0 |     2.0 |     2.0 | None                  |         0.00 |         |
    | optimizer    | factor | SGD       |     0.0 |     3.0 |     3.0 | None                  |         9.53 | *       |
    | sgd_momentum | float  | 0.0       |     0.9 |     0.9 |     0.9 | None                  |         0.00 |         |

### 11.10.1 Show variable importance

    spot_tuner.plot_importance(threshold=0.025, filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

### 11.10.2 Get the Tuned Architecture (SPOT Results)

The architecture of the `spotPython` model can be obtained by the following code:

    from spotPython.hyperparameters.values import get_one_core_model_from_X
    X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
    model_spot = get_one_core_model_from_X(X, fun_control)
    model_spot

    Net_fashionMNIST(
      (flatten): Flatten(start_dim=1, end_dim=-1)
      (linear_relu_stack): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): ReLU()
        (2): Linear(in_features=32, out_features=16, bias=True)
        (3): ReLU()
        (4): Linear(in_features=16, out_features=10, bias=True)
      )
    )

### 11.10.3 Get Default Hyperparameters

    fc = fun_control
    fc.update({"core_model_hyper_dict":
        hyper_dict[fun_control["core_model"].__name__]})
    model_default = get_one_core_model_from_X(X_start, fun_control=fc)
    model_default

    Net_fashionMNIST(
      (flatten): Flatten(start_dim=1, end_dim=-1)
      (linear_relu_stack): Sequential(
        (0): Linear(in_features=784, out_features=32, bias=True)
        (1): ReLU()
        (2): Linear(in_features=32, out_features=32, bias=True)
        (3): ReLU()
        (4): Linear(in_features=32, out_features=10, bias=True)
      )
    )

### 11.10.4 Evaluation of the Default and the Tuned Architectures

The method `train_tuned` takes a model architecture without trained weights and trains this model with the train data. The train data is split into train and validation data. The validation data is used for early stopping. The trained model weights are saved as a dictionary.

    from spotPython.torch.traintest import train_tuned
    train_tuned(net=model_default, train_dataset=train, shuffle=True,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            device = fun_control["device"],
            show_batch_interval=1_000_000,
            path=None,
            task=fun_control["task"])

    Epoch: 1 |

    MulticlassAccuracy: 0.4062083363533020 | Loss: 2.0880768024126688 | Acc: 0.4062083333333333.
    Epoch: 2 |

    MulticlassAccuracy: 0.5142499804496765 | Loss: 1.5665228478113811 | Acc: 0.5142500000000000.
    Epoch: 3 |

    MulticlassAccuracy: 0.6118749976158142 | Loss: 1.2638726920684178 | Acc: 0.6118749999999999.
    Epoch: 4 |

    MulticlassAccuracy: 0.6279583573341370 | Loss: 1.0935734656254450 | Acc: 0.6279583333333333.
    Epoch: 5 |

    MulticlassAccuracy: 0.6449583172798157 | Loss: 0.9890156506697337 | Acc: 0.6449583333333333.
    Epoch: 6 |

    MulticlassAccuracy: 0.6657500267028809 | Loss: 0.9186533939838409 | Acc: 0.6657500000000000.
    Epoch: 7 |

    MulticlassAccuracy: 0.6685833334922791 | Loss: 0.8704562251766522 | Acc: 0.6685833333333333.
    Epoch: 8 |

    MulticlassAccuracy: 0.6899583339691162 | Loss: 0.8336114618778229 | Acc: 0.6899583333333333.
    Returned to Spot: Validation loss: 0.8336114618778229

    from spotPython.torch.traintest import test_tuned
    test_tuned(net=model_default, test_dataset=test,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            shuffle=False,
            device = fun_control["device"],
            task=fun_control["task"])

    MulticlassAccuracy: 0.6801000237464905 | Loss: 0.8459481260776520 | Acc: 0.6801000000000000.
    Final evaluation: Validation loss: 0.845948126077652
    Final evaluation: Validation metric: 0.6801000237464905
    ----------------------------------------------

    (0.845948126077652, nan, tensor(0.6801))

The following code trains the model `model_spot`. If `path` is set to a filename, e.g., `path = "model_spot_trained.pt"`, the weights of the trained model will be saved to this file.

    train_tuned(net=model_spot, train_dataset=train,
            loss_function=fun_control["loss_function"],
            metric=fun_control["metric_torch"],
            shuffle=True,
            device = fun_control["device"],
            path=None,
            task=fun_control["task"])

    Epoch: 1 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 2.255

    MulticlassAccuracy: 0.4462916553020477 | Loss: 2.0947059877018135 | Acc: 0.4462916666666666.
    Epoch: 2 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 2.031

    MulticlassAccuracy: 0.4554166793823242 | Loss: 1.8582815084060034 | Acc: 0.4554166666666667.
    Epoch: 3 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.793

    MulticlassAccuracy: 0.4664166569709778 | Loss: 1.6409407677079240 | Acc: 0.4664166666666666.
    Epoch: 4 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.587

    MulticlassAccuracy: 0.4810000061988831 | Loss: 1.4696264807830255 | Acc: 0.4810000000000000.
    Epoch: 5 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.429

    MulticlassAccuracy: 0.5023750066757202 | Loss: 1.3310805119499565 | Acc: 0.5023750000000000.
    Epoch: 6 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.303

    MulticlassAccuracy: 0.5258749723434448 | Loss: 1.2129417679471275 | Acc: 0.5258750000000000.
    Epoch: 7 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.182

    MulticlassAccuracy: 0.5571249723434448 | Loss: 1.1122925583232814 | Acc: 0.5571250000000000.
    Epoch: 8 |

    Batch: 10000. Batch Size: 2. Training Loss (running): 1.094

    MulticlassAccuracy: 0.6276666522026062 | Loss: 1.0317173832049593 | Acc: 0.6276666666666667.
    Returned to Spot: Validation loss: 1.0317173832049593

    test_tuned(net=model_spot, test_dataset=test,
                shuffle=False,
                loss_function=fun_control["loss_function"],
                metric=fun_control["metric_torch"],
                device = fun_control["device"],
                task=fun_control["task"])

    MulticlassAccuracy: 0.6208000183105469 | Loss: 1.0453401397772133 | Acc: 0.6208000000000000.
    Final evaluation: Validation loss: 1.0453401397772133
    Final evaluation: Validation metric: 0.6208000183105469
    ----------------------------------------------

    (1.0453401397772133, nan, tensor(0.6208))

### 11.10.5 Detailed Hyperparameter Plots

    filename = "./figures/" + experiment_name
    spot_tuner.plot_important_hyperparameter_contour(filename=filename)

    l1:  99.99999999999999
    optimizer:  9.532233182650076

Contour plots.

### 11.10.6 Parallel Coordinates Plot

    spot_tuner.parallel_plot()

Parallel coordinates plots

### 11.10.7 Plot all Combinations of Hyperparameters

- Warning: this may take a while.

    PLOT_ALL = False
    if PLOT_ALL:
        n = spot_tuner.k
        for i in range(n-1):
            for j in range(i+1, n):
                spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)