# 5 Sequential Parameter Optimization: Using `scipy` Optimizers

This notebook describes how different optimizers form the `scipy optimize` package can be used on the surrogate. The optimization algorithms are available from https://docs.scipy.org/doc/scipy/reference/optimize.html

    import numpy as np
    from math import inf
    from spotPython.fun.objectivefunctions import analytical
    from spotPython.spot import spot
    from scipy.optimize import shgo
    from scipy.optimize import direct
    from scipy.optimize import differential_evolution
    from scipy.optimize import dual_annealing
    from scipy.optimize import basinhopping
    import matplotlib.pyplot as plt

## 5.1 The Objective Function Branin

- The `spotPython` package provides several classes of objective functions.

- We will use an analytical objective function, i.e., a function that can be described by a (closed) formula.

- Here we will use the Branin function. The 2-dim Branin function is

  \[y = a * (x2 - b * x1**2 + c * x1 - r) ** 2 + s * (1 - t) * cos(x1) + s,\] where values of a, b, c, r, s and t are: \(a = 1, b = 5.1 / (4*pi**2), c = 5 / pi, r = 6, s = 10\) and \(t = 1 / (8*pi)\).

- It has three global minima:

  \(f(x) = 0.397887\) at \((-\pi, 12.275)\), \((\pi, 2.275)\), and \((9.42478, 2.475)\).

- Input Domain: This function is usually evaluated on the square x1 in [-5, 10] x x2 in [0, 15].

    from spotPython.fun.objectivefunctions import analytical
    lower = np.array([-5,-0])
    upper = np.array([10,15])

    fun = analytical(seed=123).fun_branin

## 5.2 The Optimizer

- Differential Evalution from the `scikit.optimize` package, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution is the default optimizer for the search on the surrogate.

- Other optimiers that are available in `spotPython`:

  - `dual_annealing`
  - `direct`
  - `shgo`
  - `basinhopping`, see https://docs.scipy.org/doc/scipy/reference/optimize.html#global-optimization.

- These can be selected as follows:

  `surrogate_control = "model_optimizer": differential_evolution`

- We will use `differential_evolution`.

- The optimizer can use `1000` evaluations. This value will be passed to the `differential_evolution` method, which has the argument `maxiter` (int). It defines the maximum number of generations over which the entire differential evolution population is evolved, see https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differential_evolution.html#scipy.optimize.differential_evolution

    spot_de = spot.Spot(fun=fun,
                       lower = lower,
                       upper = upper,
                       fun_evals = 20,
                       max_time = inf,
                       seed=125,
                       noise=False,
                       show_models= False,
                       design_control={"init_size": 10},
                       surrogate_control={"n_theta": 2,
                                          "model_optimizer": differential_evolution,
                                          "model_fun_evals": 1000,
                                          })
    spot_de.run()

    <spotPython.spot.spot.Spot at 0x104135d20>

## 5.3 Print the Results

    spot_de.print_results()

    min y: 0.39951958110619046
    x0: -3.1570201165683587
    x1: 12.289980569430284

    [['x0', -3.1570201165683587], ['x1', 12.289980569430284]]

## 5.4 Show the Progress

    spot_de.plot_progress(log_y=True)

    spot_de.surrogate.plot()

## 5.5 Exercises

### 5.5.1 `dual_annealing`

- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate

### 5.5.2 `direct`

- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate

### 5.5.3 `shgo`

- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate

### 5.5.4 `basinhopping`

- Describe the optimization algorithm
- Use the algorithm as an optimizer on the surrogate

### 5.5.5 Performance Comparison

Compare the performance and run time of the 5 different optimizers:

    * `differential_evolution`
    * `dual_annealing`
    *  `direct`
    * `shgo`
    * `basinhopping`.

The Branin function has three global minima:

- \(f(x) = 0.397887\) at
  - \((-\pi, 12.275)\),
  - \((\pi, 2.275)\), and
  - \((9.42478, 2.475)\).
- Which optima are found by the optimizers? Does the `seed` change this behavior?