# 17 HPT PyTorch Lightning: VBDP

In this tutorial, we will show how `spotPython` can be integrated into the `PyTorch` Lightning training workflow for a classification task.

Caution: Data must be downloaded manually

- Ensure that the corresponding data is available as `./data/VBDP/train.csv`.

This document refers to the latest `spotPython` version, which can be installed via pip. Alternatively, the source code can be downloaded from gitHub: https://github.com/sequential-parameter-optimization/spotPython.

- Uncomment the following lines if you want to for (re-)installation the latest version of `spotPython` from GitHub.

    # import sys
    # !{sys.executable} -m pip install --upgrade build
    # !{sys.executable} -m pip install --upgrade --force-reinstall spotPython

## 17.1 Step 1: Setup

- Before we consider the detailed experimental setup, we select the parameters that affect run time, initial design size, etc.
- The parameter `MAX_TIME` specifies the maximum run time in seconds.
- The parameter `INIT_SIZE` specifies the initial design size.
- The parameter `WORKERS` specifies the number of workers.
- The prefix `PREFIX` is used for the experiment name and the name of the log file.

    MAX_TIME = 1
    INIT_SIZE = 5
    WORKERS = 0
    PREFIX="31"

    import os
    if not os.path.exists('./figures'):
        os.makedirs('./figures')

Caution: Run time and initial design size should be increased for real experiments

- `MAX_TIME` is set to one minute for demonstration purposes. For real experiments, this should be increased to at least 1 hour.
- `INIT_SIZE` is set to 5 for demonstration purposes. For real experiments, this should be increased to at least 10.
- `WORKERS` is set to 0 for demonstration purposes. For real experiments, this should be increased. See the warnings that are printed when the number of workers is set to 0.

Note: Device selection

- Although there are no .cuda() or .to(device) calls required, because Lightning does these for you, see LIGHTNINGMODULE, we would like to know which device is used. Threrefore, we imitate the LightningModule behaviour which selects the highest device.
- The method `spotPython.utils.device.getDevice()` returns the device that is used by Lightning.

## 17.2 Step 2: Initialization of the `fun_control` Dictionary

`spotPython` uses a Python dictionary for storing the information required for the hyperparameter tuning process, which was described in Section 12.2, see Initialization of the fun_control Dictionary in the documentation.

    from spotPython.utils.init import fun_control_init
    from spotPython.utils.file import get_experiment_name
    from spotPython.utils.device import getDevice

    experiment_name = get_experiment_name(prefix=PREFIX)
    fun_control = fun_control_init(
        num_workers=WORKERS,
        device=getDevice(),
        _L_in=64,
        _L_out=11)

    fun_control["device"]

    'mps'

## 17.3 Step 3: PyTorch Data Loading

### 17.3.1 Lightning Dataset and DataModule

The data loading and preprocessing is handled by `Lightning` and `PyTorch`. It comprehends the following classes:

- `CSVDataset`: A class that loads the data from a CSV file. [SOURCE]
- `CSVDataModule`: A class that prepares the data for training and testing. [SOURCE]

Section Section 17.13.1 illustrates how to access the data.

## 17.4 Step 4: Preprocessing

Preprocessing is handled by `Lightning` and `PyTorch`. It can be implemented in the `CSVDataModule` class [SOURCE] and is described in the LIGHTNINGDATAMODULE documentation. Here you can find information about the `transforms` methods.

## 17.5 Step 5: Select the NN Model (`algorithm`) and `core_model_hyper_dict`

`spotPython` includes the `NetLightBase` class [SOURCE] for configurable neural networks. The class is imported here. It inherits from the class `Lightning.LightningModule`, which is the base class for all models in `Lightning`. `Lightning.LightningModule` is a subclass of `torch.nn.Module` and provides additional functionality for the training and testing of neural networks. The class `Lightning.LightningModule` is described in the Lightning documentation.

- Here we simply add the NN Model to the fun_control dictionary by calling the function `add_core_model_to_fun_control`:

    from spotPython.light.netlightbase import NetLightBase
    from spotPython.data.light_hyper_dict import LightHyperDict
    from spotPython.hyperparameters.values import add_core_model_to_fun_control
    add_core_model_to_fun_control(core_model=NetLightBase,
                                  fun_control=fun_control,
                                  hyper_dict= LightHyperDict)

The `NetLightBase` is a configurable neural network. The hyperparameters of the model are specified in the `core_model_hyper_dict` dictionary [SOURCE].

## 17.6 Step 6: Modify `hyper_dict` Hyperparameters for the Selected Algorithm aka `core_model`

`spotPython` provides functions for modifying the hyperparameters, their bounds and factors as well as for activating and de-activating hyperparameters without re-compilation of the Python source code. These functions were described in Section 12.6.

Caution: Small number of epochs for demonstration purposes

- `epochs` and `patience` are set to small values for demonstration purposes. These values are too small for a real application.
- More resonable values are, e.g.:
  - `modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[7, 9])` and
  - `modify_hyper_parameter_bounds(fun_control, "patience", bounds=[2, 7])`

    from spotPython.hyperparameters.values import modify_hyper_parameter_bounds

    modify_hyper_parameter_bounds(fun_control, "l1", bounds=[6,13])
    modify_hyper_parameter_bounds(fun_control, "epochs", bounds=[6,13])
    modify_hyper_parameter_bounds(fun_control, "batch_size", bounds=[2, 8])

    from spotPython.hyperparameters.values import modify_hyper_parameter_levels
    modify_hyper_parameter_levels(fun_control, "optimizer",["Adam", "AdamW", "Adamax", "NAdam"])
    # modify_hyper_parameter_levels(fun_control, "optimizer", ["Adam"])

The updated `fun_control` dictionary is shown below.

    fun_control["core_model_hyper_dict"]

    {'l1': {'type': 'int',
      'default': 3,
      'transform': 'transform_power_2_int',
      'lower': 6,
      'upper': 13},
     'epochs': {'type': 'int',
      'default': 4,
      'transform': 'transform_power_2_int',
      'lower': 6,
      'upper': 13},
     'batch_size': {'type': 'int',
      'default': 4,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 8},
     'act_fn': {'levels': ['Sigmoid', 'Tanh', 'ReLU', 'LeakyReLU', 'ELU', 'Swish'],
      'type': 'factor',
      'default': 'ReLU',
      'transform': 'None',
      'class_name': 'spotPython.torch.activation',
      'core_model_parameter_type': 'instance()',
      'lower': 0,
      'upper': 5},
     'optimizer': {'levels': ['Adam', 'AdamW', 'Adamax', 'NAdam'],
      'type': 'factor',
      'default': 'SGD',
      'transform': 'None',
      'class_name': 'torch.optim',
      'core_model_parameter_type': 'str',
      'lower': 0,
      'upper': 3},
     'dropout_prob': {'type': 'float',
      'default': 0.01,
      'transform': 'None',
      'lower': 0.0,
      'upper': 0.25},
     'lr_mult': {'type': 'float',
      'default': 1.0,
      'transform': 'None',
      'lower': 0.1,
      'upper': 10.0},
     'patience': {'type': 'int',
      'default': 2,
      'transform': 'transform_power_2_int',
      'lower': 2,
      'upper': 6},
     'initialization': {'levels': ['Default', 'Kaiming', 'Xavier'],
      'type': 'factor',
      'default': 'Default',
      'transform': 'None',
      'core_model_parameter_type': 'str',
      'lower': 0,
      'upper': 2}}

## 17.7 Step 7: Data Splitting, the Objective (Loss) Function and the Metric

### 17.7.1 Evaluation

The evaluation procedure requires the specification of two elements:

1. the way how the data is split into a train and a test set (see Section 12.7.1)
2. the loss function (and a metric).

Caution: Data Splitting in Lightning

- The data splitting is handled by `Lightning`.

### 17.7.2 Loss Functions and Metrics

The loss function is specified in the configurable network class [SOURCE] We will use CrossEntropy loss for the multiclass-classification task.

### 17.7.3 Metric

- We will use the MAP@k metric [SOURCE] for the evaluation of the model. Here is an example how this metric is calculated.

    from spotPython.torch.mapk import MAPK
    import torch
    mapk = MAPK(k=2)
    target = torch.tensor([0, 1, 2, 2])
    preds = torch.tensor(
        [
            [0.5, 0.2, 0.2],  # 0 is in top 2
            [0.3, 0.4, 0.2],  # 1 is in top 2
            [0.2, 0.4, 0.3],  # 2 is in top 2
            [0.7, 0.2, 0.1],  # 2 isn't in top 2
        ]
    )
    mapk.update(preds, target)
    print(mapk.compute()) # tensor(0.6250)

    tensor(0.6250)

Similar to the loss function, the metric is specified in the configurable network class [SOURCE].

Caution: Loss Function and Metric in Lightning

- The loss function and the metric are not hyperparameters that can be tuned with `spotPython`.
- They are handled by `Lightning`.

## 17.8 Step 8: Calling the SPOT Function

### 17.8.1 Preparing the SPOT Call

The following code passes the information about the parameter ranges and bounds to `spot`. It extracts the variable types, names, and bounds

    from spotPython.hyperparameters.values import (get_bound_values,
        get_var_name,
        get_var_type,)
    var_type = get_var_type(fun_control)
    var_name = get_var_name(fun_control)
    lower = get_bound_values(fun_control, "lower")
    upper = get_bound_values(fun_control, "upper")

Now, the dictionary `fun_control` contains all information needed for the hyperparameter tuning. Before the hyperparameter tuning is started, it is recommended to take a look at the experimental design. The method `gen_design_table` [SOURCE] generates a design table as follows:

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control))

    | name           | type   | default   |   lower |   upper | transform             |
    |----------------|--------|-----------|---------|---------|-----------------------|
    | l1             | int    | 3         |     6   |   13    | transform_power_2_int |
    | epochs         | int    | 4         |     6   |   13    | transform_power_2_int |
    | batch_size     | int    | 4         |     2   |    8    | transform_power_2_int |
    | act_fn         | factor | ReLU      |     0   |    5    | None                  |
    | optimizer      | factor | SGD       |     0   |    3    | None                  |
    | dropout_prob   | float  | 0.01      |     0   |    0.25 | None                  |
    | lr_mult        | float  | 1.0       |     0.1 |   10    | None                  |
    | patience       | int    | 2         |     2   |    6    | transform_power_2_int |
    | initialization | factor | Default   |     0   |    2    | None                  |

This allows to check if all information is available and if the information is correct.

### 17.8.2 The Objective Function `fun`

The objective function `fun` from the class `HyperLight` [SOURCE] is selected next. It implements an interface from `PyTorch`’s training, validation, and testing methods to `spotPython`.

    from spotPython.fun.hyperlight import HyperLight
    fun = HyperLight().fun

### 17.8.3 Starting the Hyperparameter Tuning

The `spotPython` hyperparameter tuning is started by calling the `Spot` function [SOURCE] as described in Section 12.8.4.

    import numpy as np
    from spotPython.spot import spot
    from math import inf
    spot_tuner = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = inf,
                       max_time = MAX_TIME,
                       tolerance_x = np.sqrt(np.spacing(1)),
                       var_type = var_type,
                       var_name = var_name,
                       show_progress= True,
                       fun_control = fun_control,
                       design_control={"init_size": INIT_SIZE},
                       surrogate_control={"noise": True,
                                          "min_theta": -4,
                                          "max_theta": 3,
                                          "n_theta": len(var_name),
                                          "model_fun_evals": 10_000,
                                          })
    spot_tuner.run()

    config: {'l1': 4096, 'epochs': 4096, 'batch_size': 32, 'act_fn': ReLU(), 'optimizer': 'AdamW', 'dropout_prob': 0.10939527466721133, 'lr_mult': 4.211776903906428, 'patience': 16, 'initialization': 'Default'}
    _L_in: 64
    _L_out: 11

    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=4096, bias=True)
        (1): ReLU()
        (2): Dropout(p=0.10939527466721133, inplace=False)
        (3): Linear(in_features=4096, out_features=2048, bias=True)
        (4): ReLU()
        (5): Dropout(p=0.10939527466721133, inplace=False)
        (6): Linear(in_features=2048, out_features=2048, bias=True)
        (7): ReLU()
        (8): Dropout(p=0.10939527466721133, inplace=False)
        (9): Linear(in_features=2048, out_features=1024, bias=True)
        (10): ReLU()
        (11): Dropout(p=0.10939527466721133, inplace=False)
        (12): Linear(in_features=1024, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.465301990509033     │
    │          val_acc          │    0.07773851603269577    │
    │         val_loss          │     2.465301990509033     │
    │        valid_mapk         │    0.1492198258638382     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.1492198258638382, 'val_loss': 2.465301990509033, 'val_acc': 0.07773851603269577, 'hp_metric': 2.465301990509033}

    config: {'l1': 64, 'epochs': 128, 'batch_size': 256, 'act_fn': LeakyReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.012926647388264517, 'lr_mult': 0.832718394912432, 'patience': 8, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): LeakyReLU()
        (2): Dropout(p=0.012926647388264517, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): LeakyReLU()
        (5): Dropout(p=0.012926647388264517, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): LeakyReLU()
        (8): Dropout(p=0.012926647388264517, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): LeakyReLU()
        (11): Dropout(p=0.012926647388264517, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2949275970458984     │
    │          val_acc          │    0.24028268456459045    │
    │         val_loss          │    2.2949275970458984     │
    │        valid_mapk         │    0.36091822385787964    │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.36091822385787964, 'val_loss': 2.2949275970458984, 'val_acc': 0.24028268456459045, 'hp_metric': 2.2949275970458984}

    config: {'l1': 1024, 'epochs': 256, 'batch_size': 8, 'act_fn': Swish(), 'optimizer': 'NAdam', 'dropout_prob': 0.22086376796923401, 'lr_mult': 7.65501078489161, 'patience': 64, 'initialization': 'Xavier'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=1024, bias=True)
        (1): Swish()
        (2): Dropout(p=0.22086376796923401, inplace=False)
        (3): Linear(in_features=1024, out_features=512, bias=True)
        (4): Swish()
        (5): Dropout(p=0.22086376796923401, inplace=False)
        (6): Linear(in_features=512, out_features=512, bias=True)
        (7): Swish()
        (8): Dropout(p=0.22086376796923401, inplace=False)
        (9): Linear(in_features=512, out_features=256, bias=True)
        (10): Swish()
        (11): Dropout(p=0.22086376796923401, inplace=False)
        (12): Linear(in_features=256, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.408764600753784     │
    │          val_acc          │    0.1342756152153015     │
    │         val_loss          │     2.408764600753784     │
    │        valid_mapk         │     0.213541641831398     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.213541641831398, 'val_loss': 2.408764600753784, 'val_acc': 0.1342756152153015, 'hp_metric': 2.408764600753784}

    config: {'l1': 512, 'epochs': 512, 'batch_size': 16, 'act_fn': Sigmoid(), 'optimizer': 'Adam', 'dropout_prob': 0.1890928563375006, 'lr_mult': 2.3450676871382794, 'patience': 32, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=512, bias=True)
        (1): Sigmoid()
        (2): Dropout(p=0.1890928563375006, inplace=False)
        (3): Linear(in_features=512, out_features=256, bias=True)
        (4): Sigmoid()
        (5): Dropout(p=0.1890928563375006, inplace=False)
        (6): Linear(in_features=256, out_features=256, bias=True)
        (7): Sigmoid()
        (8): Dropout(p=0.1890928563375006, inplace=False)
        (9): Linear(in_features=256, out_features=128, bias=True)
        (10): Sigmoid()
        (11): Dropout(p=0.1890928563375006, inplace=False)
        (12): Linear(in_features=128, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.296860933303833     │
    │          val_acc          │    0.24381625652313232    │
    │         val_loss          │     2.296860933303833     │
    │        valid_mapk         │    0.3250736594200134     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3250736594200134, 'val_loss': 2.296860933303833, 'val_acc': 0.24381625652313232, 'hp_metric': 2.296860933303833}

    config: {'l1': 256, 'epochs': 4096, 'batch_size': 64, 'act_fn': ReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.0708380794924471, 'lr_mult': 9.528945328733357, 'patience': 4, 'initialization': 'Xavier'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=256, bias=True)
        (1): ReLU()
        (2): Dropout(p=0.0708380794924471, inplace=False)
        (3): Linear(in_features=256, out_features=128, bias=True)
        (4): ReLU()
        (5): Dropout(p=0.0708380794924471, inplace=False)
        (6): Linear(in_features=128, out_features=128, bias=True)
        (7): ReLU()
        (8): Dropout(p=0.0708380794924471, inplace=False)
        (9): Linear(in_features=128, out_features=64, bias=True)
        (10): ReLU()
        (11): Dropout(p=0.0708380794924471, inplace=False)
        (12): Linear(in_features=64, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.3139610290527344     │
    │          val_acc          │    0.21554769575595856    │
    │         val_loss          │    2.3139610290527344     │
    │        valid_mapk         │    0.2888889014720917     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.2888889014720917, 'val_loss': 2.3139610290527344, 'val_acc': 0.21554769575595856, 'hp_metric': 2.3139610290527344}

    config: {'l1': 64, 'epochs': 128, 'batch_size': 128, 'act_fn': ELU(), 'optimizer': 'Adam', 'dropout_prob': 0.12309417501765757, 'lr_mult': 0.8448940317070717, 'patience': 16, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2695024013519287     │
    │          val_acc          │    0.25441697239875793    │
    │         val_loss          │    2.2695024013519287     │
    │        valid_mapk         │    0.40205439925193787    │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.40205439925193787, 'val_loss': 2.2695024013519287, 'val_acc': 0.25441697239875793, 'hp_metric': 2.2695024013519287}

    spotPython tuning: 2.2695024013519287 [#---------] 10.36%

    config: {'l1': 64, 'epochs': 128, 'batch_size': 128, 'act_fn': Sigmoid(), 'optimizer': 'Adam', 'dropout_prob': 0.08465354887179713, 'lr_mult': 1.0050301840926235, 'patience': 16, 'initialization': 'Default'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): Sigmoid()
        (2): Dropout(p=0.08465354887179713, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): Sigmoid()
        (5): Dropout(p=0.08465354887179713, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): Sigmoid()
        (8): Dropout(p=0.08465354887179713, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): Sigmoid()
        (11): Dropout(p=0.08465354887179713, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.3968088626861572     │
    │          val_acc          │    0.09540636092424393    │
    │         val_loss          │    2.3968088626861572     │
    │        valid_mapk         │    0.1987365037202835     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.1987365037202835, 'val_loss': 2.3968088626861572, 'val_acc': 0.09540636092424393, 'hp_metric': 2.3968088626861572}

    spotPython tuning: 2.2695024013519287 [##--------] 15.14%

    config: {'l1': 2048, 'epochs': 256, 'batch_size': 64, 'act_fn': ELU(), 'optimizer': 'AdamW', 'dropout_prob': 0.20745264374014713, 'lr_mult': 0.1, 'patience': 32, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=2048, bias=True)
        (1): ELU()
        (2): Dropout(p=0.20745264374014713, inplace=False)
        (3): Linear(in_features=2048, out_features=1024, bias=True)
        (4): ELU()
        (5): Dropout(p=0.20745264374014713, inplace=False)
        (6): Linear(in_features=1024, out_features=1024, bias=True)
        (7): ELU()
        (8): Dropout(p=0.20745264374014713, inplace=False)
        (9): Linear(in_features=1024, out_features=512, bias=True)
        (10): ELU()
        (11): Dropout(p=0.20745264374014713, inplace=False)
        (12): Linear(in_features=512, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.273996114730835     │
    │          val_acc          │    0.25441697239875793    │
    │         val_loss          │     2.273996114730835     │
    │        valid_mapk         │    0.3512345552444458     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3512345552444458, 'val_loss': 2.273996114730835, 'val_acc': 0.25441697239875793, 'hp_metric': 2.273996114730835}

    spotPython tuning: 2.2695024013519287 [###-------] 28.45%

    config: {'l1': 128, 'epochs': 64, 'batch_size': 128, 'act_fn': ELU(), 'optimizer': 'AdamW', 'dropout_prob': 0.25, 'lr_mult': 1.390967859401377, 'patience': 32, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=128, bias=True)
        (1): ELU()
        (2): Dropout(p=0.25, inplace=False)
        (3): Linear(in_features=128, out_features=64, bias=True)
        (4): ELU()
        (5): Dropout(p=0.25, inplace=False)
        (6): Linear(in_features=64, out_features=64, bias=True)
        (7): ELU()
        (8): Dropout(p=0.25, inplace=False)
        (9): Linear(in_features=64, out_features=32, bias=True)
        (10): ELU()
        (11): Dropout(p=0.25, inplace=False)
        (12): Linear(in_features=32, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.3007595539093018     │
    │          val_acc          │    0.21554769575595856    │
    │         val_loss          │    2.3007595539093018     │
    │        valid_mapk         │    0.3140271306037903     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3140271306037903, 'val_loss': 2.3007595539093018, 'val_acc': 0.21554769575595856, 'hp_metric': 2.3007595539093018}

    spotPython tuning: 2.2695024013519287 [###-------] 34.99%

    config: {'l1': 4096, 'epochs': 128, 'batch_size': 4, 'act_fn': ReLU(), 'optimizer': 'Adamax', 'dropout_prob': 0.13486862932590807, 'lr_mult': 0.1, 'patience': 8, 'initialization': 'Kaiming'}
    _L_in: 64
    _L_out: 11

    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=4096, bias=True)
        (1): ReLU()
        (2): Dropout(p=0.13486862932590807, inplace=False)
        (3): Linear(in_features=4096, out_features=2048, bias=True)
        (4): ReLU()
        (5): Dropout(p=0.13486862932590807, inplace=False)
        (6): Linear(in_features=2048, out_features=2048, bias=True)
        (7): ReLU()
        (8): Dropout(p=0.13486862932590807, inplace=False)
        (9): Linear(in_features=2048, out_features=1024, bias=True)
        (10): ReLU()
        (11): Dropout(p=0.13486862932590807, inplace=False)
        (12): Linear(in_features=1024, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.286646842956543     │
    │          val_acc          │    0.25088340044021606    │
    │         val_loss          │     2.286646842956543     │
    │        valid_mapk         │    0.3474178910255432     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3474178910255432, 'val_loss': 2.286646842956543, 'val_acc': 0.25088340044021606, 'hp_metric': 2.286646842956543}

    spotPython tuning: 2.2695024013519287 [##########] 100.00% Done...

    <spotPython.spot.spot.Spot at 0x2bb196c50>

## 17.9 Step 9: Tensorboard

The textual output shown in the console (or code cell) can be visualized with Tensorboard.

    tensorboard --logdir=runs/lightning_logs

Further information can be found in the PyTorch Lightning documentation for Tensorboard.

## 17.10 Step 10: Results

After the hyperparameter tuning run is finished, the results can be analyzed as described in Section 12.10.

    spot_tuner.plot_progress(log_y=False,
        filename="./figures/" + experiment_name+"_progress.png")

Progress plot. Black dots denote results from the initial design. Red dots illustrate the improvement found by the surrogate model based optimization.

    from spotPython.utils.eda import gen_design_table
    print(gen_design_table(fun_control=fun_control, spot=spot_tuner))

    | name           | type   | default   |   lower |   upper |               tuned | transform             |   importance | stars   |
    |----------------|--------|-----------|---------|---------|---------------------|-----------------------|--------------|---------|
    | l1             | int    | 3         |     6.0 |    13.0 |                 6.0 | transform_power_2_int |         0.00 |         |
    | epochs         | int    | 4         |     6.0 |    13.0 |                 7.0 | transform_power_2_int |         0.00 |         |
    | batch_size     | int    | 4         |     2.0 |     8.0 |                 7.0 | transform_power_2_int |         0.00 |         |
    | act_fn         | factor | ReLU      |     0.0 |     5.0 |                 4.0 | None                  |         0.00 |         |
    | optimizer      | factor | SGD       |     0.0 |     3.0 |                 0.0 | None                  |         3.76 | *       |
    | dropout_prob   | float  | 0.01      |     0.0 |    0.25 | 0.12309417501765757 | None                  |         0.00 |         |
    | lr_mult        | float  | 1.0       |     0.1 |    10.0 |  0.8448940317070717 | None                  |         0.18 | .       |
    | patience       | int    | 2         |     2.0 |     6.0 |                 4.0 | transform_power_2_int |         0.03 |         |
    | initialization | factor | Default   |     0.0 |     2.0 |                 1.0 | None                  |       100.00 | ***     |

    spot_tuner.plot_importance(threshold=0.025,
        filename="./figures/" + experiment_name+"_importance.png")

Variable importance plot, threshold 0.025.

### 17.10.1 Get the Tuned Architecture

    from spotPython.light.utils import get_tuned_architecture
    config = get_tuned_architecture(spot_tuner, fun_control)

- Test on the full data set

    from spotPython.light.traintest import test_model
    test_model(config, fun_control)

    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃        Test metric        ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.193178176879883     │
    │      test_mapk_epoch      │    0.4349476397037506     │
    │          val_acc          │    0.35077792406082153    │
    │         val_loss          │     2.193178176879883     │
    └───────────────────────────┴───────────────────────────┘

    test_model result: {'test_mapk_epoch': 0.4349476397037506, 'val_loss': 2.193178176879883, 'val_acc': 0.35077792406082153, 'hp_metric': 2.193178176879883}

    (2.193178176879883, 0.35077792406082153)

    from spotPython.light.traintest import load_light_from_checkpoint

    model_loaded = load_light_from_checkpoint(config, fun_control)

    Loading model from runs/lightning_logs/64_128_128_ELU()_Adam_0.12309417501765757_0.8448940317070717_16_Kaiming_TEST/checkpoints/last.ckpt

### 17.10.2 Cross Validation With Lightning

- The `KFold` class from `sklearn.model_selection` is used to generate the folds for cross-validation.
- These mechanism is used to generate the folds for the final evaluation of the model.
- The `CrossValidationDataModule` class [SOURCE] is used to generate the folds for the hyperparameter tuning process.
- It is called from the `cv_model` function [SOURCE].

    from spotPython.light.traintest import cv_model
    # set the number of folds to 10
    fun_control["k_folds"] = 10
    cv_model(config, fun_control)

    k: 0
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.1672003269195557     │
    │          val_acc          │    0.3802816867828369     │
    │         val_loss          │    2.1672003269195557     │
    │        valid_mapk         │    0.44366195797920227    │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.44366195797920227, 'val_loss': 2.1672003269195557, 'val_acc': 0.3802816867828369, 'hp_metric': 2.1672003269195557}
    k: 1
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.1558444499969482     │
    │          val_acc          │    0.3802816867828369     │
    │         val_loss          │    2.1558444499969482     │
    │        valid_mapk         │    0.4624413251876831     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.4624413251876831, 'val_loss': 2.1558444499969482, 'val_acc': 0.3802816867828369, 'hp_metric': 2.1558444499969482}
    k: 2
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.3069982528686523     │
    │          val_acc          │    0.2112676054239273     │
    │         val_loss          │    2.3069982528686523     │
    │        valid_mapk         │    0.26995304226875305    │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.26995304226875305, 'val_loss': 2.3069982528686523, 'val_acc': 0.2112676054239273, 'hp_metric': 2.3069982528686523}
    k: 3
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2378978729248047     │
    │          val_acc          │    0.28169015049934387    │
    │         val_loss          │    2.2378978729248047     │
    │        valid_mapk         │    0.3802816867828369     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3802816867828369, 'val_loss': 2.2378978729248047, 'val_acc': 0.28169015049934387, 'hp_metric': 2.2378978729248047}
    k: 4
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.273291826248169     │
    │          val_acc          │    0.23943662643432617    │
    │         val_loss          │     2.273291826248169     │
    │        valid_mapk         │    0.3380281627178192     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3380281627178192, 'val_loss': 2.273291826248169, 'val_acc': 0.23943662643432617, 'hp_metric': 2.273291826248169}
    k: 5
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │     2.299381732940674     │
    │          val_acc          │    0.22535210847854614    │
    │         val_loss          │     2.299381732940674     │
    │        valid_mapk         │    0.2863849699497223     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.2863849699497223, 'val_loss': 2.299381732940674, 'val_acc': 0.22535210847854614, 'hp_metric': 2.299381732940674}
    k: 6
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 636
    Val Dataset Size: 71

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2793431282043457     │
    │          val_acc          │    0.2535211145877838     │
    │         val_loss          │    2.2793431282043457     │
    │        valid_mapk         │    0.35680750012397766    │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.35680750012397766, 'val_loss': 2.2793431282043457, 'val_acc': 0.2535211145877838, 'hp_metric': 2.2793431282043457}
    k: 7
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 637
    Val Dataset Size: 70

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2517879009246826     │
    │          val_acc          │    0.27142858505249023    │
    │         val_loss          │    2.2517879009246826     │
    │        valid_mapk         │    0.3571428656578064     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.3571428656578064, 'val_loss': 2.2517879009246826, 'val_acc': 0.27142858505249023, 'hp_metric': 2.2517879009246826}
    k: 8
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 637
    Val Dataset Size: 70

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2574641704559326     │
    │          val_acc          │    0.27142858505249023    │
    │         val_loss          │    2.2574641704559326     │
    │        valid_mapk         │     0.380952388048172     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.380952388048172, 'val_loss': 2.2574641704559326, 'val_acc': 0.27142858505249023, 'hp_metric': 2.2574641704559326}
    k: 9
    model: NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )
    Train Dataset Size: 637
    Val Dataset Size: 70

    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
    ┃      Validate metric      ┃       DataLoader 0        ┃
    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
    │         hp_metric         │    2.2125566005706787     │
    │          val_acc          │    0.3142857253551483     │
    │         val_loss          │    2.2125566005706787     │
    │        valid_mapk         │    0.4119047522544861     │
    └───────────────────────────┴───────────────────────────┘

    train_model result: {'valid_mapk': 0.4119047522544861, 'val_loss': 2.2125566005706787, 'val_acc': 0.3142857253551483, 'hp_metric': 2.2125566005706787}
    cv_model mapk result: 0.3687558650970459

    0.3687558650970459

Note: Evaluation for the Final Comaprison

- This is the evaluation that will be used in the comparison.

### 17.10.3 Detailed Hyperparameter Plots

    filename = "./figures/" + experiment_name
    spot_tuner.plot_important_hyperparameter_contour(filename=filename)

    optimizer:  3.7556982291542074
    lr_mult:  0.181440357517405
    patience:  0.03481620409374387
    initialization:  100.0

Contour plots.

### 17.10.4 Parallel Coordinates Plot

    spot_tuner.parallel_plot()

Parallel coordinates plots

### 17.10.5 Plot all Combinations of Hyperparameters

- Warning: this may take a while.

    PLOT_ALL = False
    if PLOT_ALL:
        n = spot_tuner.k
        for i in range(n-1):
            for j in range(i+1, n):
                spot_tuner.plot_contour(i=i, j=j, min_z=min_z, max_z = max_z)

### 17.10.6 Visualizing the Activation Distribution

Reference:

- The following code is based on [PyTorch Lightning TUTORIAL 2: ACTIVATION FUNCTIONS], Author: Phillip Lippe, License: [CC BY-SA], Generated: 2023-03-15T09:52:39.179933.

After we have trained the models, we can look at the actual activation values that find inside the model. For instance, how many neurons are set to zero in ReLU? Where do we find most values in Tanh? To answer these questions, we can write a simple function which takes a trained model, applies it to a batch of images, and plots the histogram of the activations inside the network:

    from spotPython.torch.activation import Sigmoid, Tanh, ReLU, LeakyReLU, ELU, Swish
    act_fn_by_name = {"sigmoid": Sigmoid, "tanh": Tanh, "relu": ReLU, "leakyrelu": LeakyReLU, "elu": ELU, "swish": Swish}

    from spotPython.hyperparameters.values import get_one_config_from_X
    X = spot_tuner.to_all_dim(spot_tuner.min_X.reshape(1,-1))
    config = get_one_config_from_X(X, fun_control)
    model = fun_control["core_model"](**config, _L_in=64, _L_out=11)
    model

    NetLightBase(
      (train_mapk): MAPK()
      (valid_mapk): MAPK()
      (test_mapk): MAPK()
      (layers): Sequential(
        (0): Linear(in_features=64, out_features=64, bias=True)
        (1): ELU()
        (2): Dropout(p=0.12309417501765757, inplace=False)
        (3): Linear(in_features=64, out_features=32, bias=True)
        (4): ELU()
        (5): Dropout(p=0.12309417501765757, inplace=False)
        (6): Linear(in_features=32, out_features=32, bias=True)
        (7): ELU()
        (8): Dropout(p=0.12309417501765757, inplace=False)
        (9): Linear(in_features=32, out_features=16, bias=True)
        (10): ELU()
        (11): Dropout(p=0.12309417501765757, inplace=False)
        (12): Linear(in_features=16, out_features=11, bias=True)
      )
    )

    from spotPython.utils.eda import visualize_activations
    visualize_activations(model, color=f"C{0}")

## 17.11 Submission

    import numpy as np
    import pandas as pd
    from sklearn.preprocessing import OrdinalEncoder

    import pandas as pd
    from sklearn.preprocessing import OrdinalEncoder
    train_df = pd.read_csv('./data/VBDP/train.csv', index_col=0)
    # remove the id column
    # train_df = train_df.drop(columns=['id'])
    n_samples = train_df.shape[0]
    n_features = train_df.shape[1] - 1
    target_column = "prognosis"
    # Encode our prognosis labels as integers for easier decoding later
    enc = OrdinalEncoder()
    y = enc.fit_transform(train_df[[target_column]])
    test_df = pd.read_csv('./data/VBDP/test.csv', index_col=0)
    test_df

| sudden_fever | headache | mouth_bleed | nose_bleed | muscle_pain | joint_pain | vomiting | rash | diarrhea | hypotension | ... | lymph_swells | breathing_restriction | toe_inflammation | finger_inflammation | lips_irritation | itchiness | ulcers | toenail_loss | speech_problem | bullseye_rash
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
id |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
707 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
708 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
709 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0
710 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
711 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ...
1005 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
1006 | 1.0 | 0.0 | 1.0 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
1007 | 1.0 | 0.0 | 0.0 | 1.0 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 1.0 | ... | 1.0 | 1.0 | 1.0 | 1.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
1008 | 1.0 | 0.0 | 1.0 | 1.0 | 1.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
1009 | 1.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.0 | 0.0 | 1.0 | 0.0 | ... | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0

303 rows × 64 columns

    X_tensor = torch.Tensor(test_df.values)
    X_tensor = X_tensor.to(fun_control["device"])

    y = model_loaded(X_tensor)
    y.shape

    torch.Size([303, 11])

    # convert the predictions to a numpy array
    y = y.cpu().detach().numpy()
    y

    array([[3.8623641e-04, 5.4584861e-02, 1.6170354e-01, ..., 2.7317306e-01,
            3.1393211e-02, 3.9235082e-01],
           [9.7857934e-01, 7.9985512e-03, 8.7480340e-04, ..., 8.0592511e-04,
            1.0031158e-03, 2.1533536e-04],
           [1.3108379e-02, 6.6869496e-03, 6.3017502e-02, ..., 8.9552659e-01,
            7.3428885e-03, 6.2888633e-03],
           ...,
           [5.9503631e-04, 4.4238575e-02, 6.9695227e-03, ..., 2.1993993e-02,
            7.7453875e-03, 4.6672658e-03],
           [7.2374911e-04, 2.2853527e-03, 1.6974222e-03, ..., 9.8508954e-01,
            9.6827315e-04, 4.7092509e-04],
           [2.0671294e-03, 8.5416120e-03, 6.4494914e-01, ..., 2.7699485e-02,
            8.6674727e-02, 1.8409789e-01]], dtype=float32)

    test_sorted_prediction_ids = np.argsort(-y, axis=1)
    test_top_3_prediction_ids = test_sorted_prediction_ids[:,:3]
    original_shape = test_top_3_prediction_ids.shape
    test_top_3_prediction = enc.inverse_transform(test_top_3_prediction_ids.reshape(-1, 1))
    test_top_3_prediction = test_top_3_prediction.reshape(original_shape)
    test_df['prognosis'] = np.apply_along_axis(lambda x: np.array(' '.join(x), dtype="object"), 1, test_top_3_prediction)
    test_df['prognosis'].reset_index().to_csv('./data/VBDP/submission.csv', index=False)

## 17.12 Appendix

### 17.12.1 Differences to the spotPython Approaches for `torch`, `sklearn` and `river`

Caution: Data Loading in Lightning

- Data loading is handled independently from the `fun_control` dictionary by `Lightning` and `PyTorch`.
- In contrast to `spotPython` with `torch`, `river` and `sklearn`, the data sets are not added to the `fun_control` dictionary.

## 17.13 Specification of the Preprocessing Model

The `fun_control` dictionary, the `torch`, `sklearn`and `river` versions of `spotPython` allow the specification of a data preprocessing pipeline, e.g., for the scaling of the data or for the one-hot encoding of categorical variables, see Section 12.4. This feature is not used in the `Lightning` version.

Caution: Data preprocessing in Lightning

Lightning allows the data preprocessing to be specified in the `LightningDataModule` class. It is not considered here, because it should be computed at one location only.

### 17.13.1 Taking a Look at the Data

    import torch
    from spotPython.light.csvdataset import CSVDataset
    from torch.utils.data import DataLoader
    from torchvision.transforms import ToTensor

    # Create an instance of CSVDataset
    dataset = CSVDataset(csv_file="./data/VBDP/train.csv", train=True)
    # show the dimensions of the input data
    print(dataset[0][0].shape)
    # show the first element of the input data
    print(dataset[0][0])
    # show the size of the dataset
    print(f"Dataset Size: {len(dataset)}")

    torch.Size([64])
    tensor([1., 1., 0., 1., 1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 1., 1., 0., 0.,
            1., 0., 1., 0., 1., 1., 1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 0., 0.,
            1., 0., 0., 0., 0., 0., 1., 0., 1., 0., 1., 0., 0., 0., 0., 1., 0., 1.,
            0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
    Dataset Size: 707

    # Set batch size for DataLoader
    batch_size = 3
    # Create DataLoader
    dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True)

    # Iterate over the data in the DataLoader
    for batch in dataloader:
        inputs, targets = batch
        print(f"Batch Size: {inputs.size(0)}")
        print("---------------")
        print(f"Inputs: {inputs}")
        print(f"Targets: {targets}")
        break

    Batch Size: 3
    ---------------
    Inputs: tensor([[1., 1., 0., 0., 1., 1., 0., 0., 1., 0., 0., 0., 0., 0., 1., 0., 0., 0.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
            [1., 1., 1., 0., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 1., 1.,
             0., 0., 1., 0., 1., 1., 1., 0., 1., 0., 1., 1., 0., 0., 1., 1., 1., 1.,
             1., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
             0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
            [1., 1., 1., 1., 1., 1., 1., 0., 0., 1., 0., 0., 1., 1., 0., 1., 1., 1.,
             1., 1., 0., 0., 1., 1., 1., 0., 1., 1., 1., 1., 0., 0., 0., 0., 0., 1.,
             1., 0., 1., 0., 0., 0., 1., 1., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0.,
             1., 0., 0., 0., 0., 1., 0., 0., 0., 0.]])
    Targets: tensor([0, 4, 3])