Sequential Model-based Algorithm Configuration

Last update: Jan 05, 2023

Overview

SMAC v3 Project

Attention: This package is a reimplementation of the original SMAC tool (see reference below). However, the reimplementation slightly differs from the original SMAC. For comparisons against the original SMAC, we refer to a stable release of SMAC (v2) in Java which can be found here.

The documentation can be found here.

Status for master branch:

Status for the development branch

OVERVIEW

SMAC is a tool for algorithm configuration to optimize the parameters of arbitrary algorithms across a set of instances. This also includes hyperparameter optimization of ML algorithms. The main core consists of Bayesian Optimization in combination with an aggressive racing mechanism to efficiently decide which of two configurations performs better.

For a detailed description of its main idea, we refer to

Hutter, F. and Hoos, H. H. and Leyton-Brown, K.
Sequential Model-Based Optimization for General Algorithm Configuration
In: Proceedings of the conference on Learning and Intelligent OptimizatioN (LION 5)

SMAC v3 is written in Python3 and continuously tested with Python 3.6 and python3.6. Its Random Forest is written in C++.

Installation

Requirements

Besides the listed requirements (see requirements.txt), the random forest used in SMAC3 requires SWIG (>= 3.0, <4.0) as a build dependency:

apt-get install swig

On Arch Linux (or any distribution with swig4 as default implementation):

pacman -Syu swig3
ln -s /usr/bin/swig-3 /usr/bin/swig

Installation via pip

SMAC3 is available on PyPI.

pip install smac

Manual Installation

git clone https://github.com/automl/SMAC3.git && cd SMAC3
cat requirements.txt | xargs -n 1 -L 1 pip install
pip install .

Installation in Anaconda

If you use Anaconda as your Python environment, you have to install three packages before you can install SMAC:

conda install gxx_linux-64 gcc_linux-64 swig

Optional dependencies

SMAC3 comes with a set of optional dependencies that can be installed using setuptools extras:

lhd: Latin hypercube design
gp: Gaussian process models

These can be installed from PyPI or manually:

# from PyPI
pip install smac[gp]

# manually
pip install .[gp,lhd]

For convenience, there is also an all meta-dependency that installs all optional dependencies:

pip install smac[all]

License

This program is free software: you can redistribute it and/or modify it under the terms of the 3-clause BSD license (please see the LICENSE file).

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

You should have received a copy of the 3-clause BSD license along with this program (see LICENSE file). If not, see https://opensource.org/licenses/BSD-3-Clause.

USAGE

The usage of SMAC v3 is mainly the same as provided with SMAC v2.08. It supports the same parameter configuration space syntax (except for extended forbidden constraints) and interface to target algorithms.

Examples

See examples/

examples/rosenbrock.py - example on how to optimize a Python function
examples/spear_qcp/run.sh - example on how to optimize the SAT solver Spear on a set of SAT formulas

Contact

SMAC3 is developed by the AutoML Group of the University of Freiburg.

If you found a bug, please report to https://github.com/automl/SMAC3/issues.

Our guidelines for contributing to this package can be found here

Comments

Unexpected keyword argument 'rng' in smac and pyrfr==0.7.2

Hi! I'm new to python and smac. I try to run smac to optimize hyperparameters for a clustering algorithm.

Consider the following code.

    smac = SMAC(scenario=my_scenario, rng=np.random.RandomState(my_seed), tae_runner=my_run)
    parameters = smac.optimize()

I use smac==0.6.0 and pynisher==0.4 as suggested here: https://github.com/automl/auto-sklearn/issues/384#issuecomment-355039165. Got the error:

Traceback (most recent call last):
. . .
  File "/nfs/home/vshalamov/WORK/MultiClustering/ThreadsB.py", line 158, in run
    self.parameters = self.smac.optimize()
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/facade/smac_facade.py", line 365, in optimize
    incumbent = self.solver.run()
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/optimizer/smbo.py", line 129, in run
    self.incumbent = self.initial_design.run()
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/initial_design/single_config_initial_design.py", line 82, in run
    "0"))
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/tae/execute_ta_run.py", line 168, in start
    instance_specific=instance_specific)
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/tae/execute_func.py", line 156, in run
    runtime = float(obj.wall_clock_time)
AttributeError: 'function_wrapper' object has no attribute 'wall_clock_time'

I found the following code in the smac. Seems, that enforce_limit produces an object without wall_clock_time attribute:

if self.use_pynisher:
    obj = pynisher.enforce_limits(**arguments)(self.ta)
    . . .
    runtime = float(obj.wall_clock_time)

I tried to upgrade pynisher to 0.4.1 or 0.4.2, according to smac 6.0 requirements: https://github.com/automl/SMAC3/blob/9b8524a9b8011f88231e7cc911b3f198d6f3fd70/requirements.txt

After that, I started getting another error:

  . . .
  File "/nfs/home/vshalamov/WORK/MultiClustering/ThreadsEx.py", line 63, in run
    self.parameters = smac.optimize()
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/facade/smac_facade.py", line 365, in optimize
    incumbent = self.solver.run()
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/optimizer/smbo.py", line 148, in run
    challengers = self.choose_next(X, Y)
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/optimizer/smbo.py", line 216, in choose_next
    self.model.train(X, Y)
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/epm/base_epm.py", line 111, in train
    return self._train(X, Y)
  File "/nfs/home/vshalamov/WORK/environment/multi-clustering-environment/lib/python3.4/site-packages/smac/epm/rf_with_instances.py", line 142, in _train
    self.rf.fit(data, rng=self.rng)
TypeError: fit() got an unexpected keyword argument 'rng'

As suggested in https://github.com/automl/auto-sklearn/issues/342 I tried different versions of pyrfr (0.5.0, 0.6.0, 0.6.1, 0.7.2) , but nothing changed.

I believe that I've satisfied all requirements for smac==0.6.0: https://github.com/automl/SMAC3/blob/9b8524a9b8011f88231e7cc911b3f198d6f3fd70/requirements.txt

pip list output:

Click to expand . . .

alabaster (0.7.10)
argparse (1.4.0)
auto-sklearn (0.2.1)
Babel (2.5.1)
certifi (2017.11.5)
chardet (3.0.4)
ConfigSpace (0.3.8)
cycler (0.10.0)
Cython (0.27.3)
docutils (0.14)
idna (2.6)
imagesize (0.7.1)
Jinja2 (2.10)
joblib (0.11)
liac-arff (2.1.1)
lockfile (0.12.2)
MarkupSafe (1.0)
matplotlib (2.1.0)
nose (1.3.7)
numpy (1.13.3)
pandas (0.21.0)
pip (1.5.4)
psutil (5.4.3)
Pygments (2.2.0)
pynisher (0.4.1)
pyparsing (2.2.0)
pyrfr (0.7.2)
python-dateutil (2.6.1)
pytz (2017.3)
PyYAML (3.12)
requests (2.18.4)
scikit-learn (0.18.2)
scipy (1.0.0)
setuptools (2.2)
six (1.11.0)
sklearn (0.0)
smac (0.6.0)
snowballstemmer (1.2.1)
Sphinx (1.6.5)
sphinx-rtd-theme (0.2.4)
sphinxcontrib-websupport (1.0.1)
typing (3.6.2)
urllib3 (1.22)

Any idea what is happening?

bug documentation

opened by sslavian812 27

MAINT documentation
Rewriting documentation in general.

replacing example leading-ones with svm

add options-overview

refactor navigation structure (separating manual and quickstart)

This is not to be merged right away, but I would greatly appreciate an opinion on the work so far to see whether it is going in the right direction. Also to know things you are missing in the documentation in general would help.

To read, with sphinx installed:

cd doc/ make html
opened by shukon 22
Integration of new pyrfr

I replaced the old pyrfr with the new one. Locally all the unittests run through without errors or failures, which should still be the case when the new pyrfr version is on pypi

opened by AndreBiedenkapp 20
PiBO
PiBO - BO with user Priors over the Optimum

New attempt on PiBO - here are the changes from last time:

New Added unit tests for PriorAcquisitionFunction (and required supporting classes)

Tests for the _compute_prior for different shapes

Tests for standard computation for both EI (does not require rescaling) and TS (does require rescaling)

Tests for discretizing the prior (relevant for random forests

Added unit test for LocalAndSortedPriorRandomSearch

Since this one is pretty much tested (it just appends two different random searches to one another), there is really only one thing to test to me, but I may be wrong

Removed the need to sample the prior to get the co-domain of the prior

Moved to closed-form in ConfigSpace

Changed input kwargs in facade

Now called user_prior_kwargs (never got acquisition_function_kwargs to work, since the underlying acquisition function gets arguments it does not expect, like decay_beta)

Confirmed issue combining inactive with Normal / Beta hyperparameters

Due to what I believe is some old TPE support, Normal parameters are not normalized to 0-1 range (presumably since they can be used without upper and lower bounds

Inactive parameters are flagged with a -1 value (right?) for Uniform parameters, which does not mesh with the possibly unbounded nature of Normal parameters

As such, one could force Normal parameters to be bounded, normalize, and allow for PiBO with inactive parameters, but there may be some other issue regaring this that I fail to see. Simply put, I don't know if it's OK to forcibly bound and normalize Normal parameters in ConfigSpace.

Added beauitified example usage

May not be the most intuitive, but at least it showcases how to use it

And of course, right branch this time
opened by hvarfner 19
Issue/#193 restore scenario

Implementing state-restoration for smac-cmdline as mentioned in issue #193. simply use the --restore_state <FOLDER>-option. Currently simply assuming for scenario to be the same, differing options can lead to unexpected behavior (except for the limits, e.g. runcount_limit, wallclock_limit, tuner-timeout).

opened by shukon 19

Examples update

Features

Makes the quickstart guide runnable by sphinx-gallry.
Makes examples also runnable by sphinx-gallery.
Adds make command make doc-with-examples which will generate examples outputs with doc, make doc will just generate text. Also include the API building portion and the linkchecking portion of doc building.

Important Note

Examples are only runnable in sphinx-gallery with python > 3.9, see Version Issues.

Major changes

quickstart.rst was converted to a quickstart_example.py to enable sphinx-gallery.
Examples are seperated into categories.
Examples renamed from *.py to *_example.py, unless file name was *_func.py, in which case it was unchanged.

Minor Changes

Flake 8 fixes for spear_qcp as there were a lot of complaints running pre-commit.

To build and view

$ cd SMAC3
$ make clean
$ make doc-with-examples
$ xdg-open ./doc/build/html/index.html  # Or firefox, google-chrome, etc..

Does not

Does not improve the code or text of the examples except for minor changes in quickstart_example.py

Version issues

The examples are successfully by sphinx-gallery with python 3.9.5, but this is not the case with python 3.8.6. Nothing in the python version notes indicates why this might be the case.

As this is only likely to be run by us for generating the doc and publishing it, I think this will have to do for now. I imagine this might be revisited in the future, or else live with this issue until support for python < 3.9 is dropped.

Issues

The linkchecking portion of make doc complains about many links yet manual checking seems to show all of them working correctly.
6 examples fail to run using python < 3.9.0 for unknown import errors, the log of which is given below

`make doc`

make -C doc html
make[1]: Entering directory '/home/skantify/code/SMAC3/doc'
sphinx-build -b html -d build/doctrees   . build/html
Running Sphinx v4.1.1
loading pickled environment... done
[autosummary] generating autosummary for: api.rst, basic_usage.rst, contact.rst, examples/SMAC4BO/SMAC4BO_mlp_example.rst, examples/SMAC4BO/SMAC4BO_rosenbrock_example.rst, examples/SMAC4BO/SMAC4BO_sgd_instances_example.rst, examples/SMAC4BO/sg_execution_times.rst, examples/SMAC4HPO/SMAC4HPO_acq_rosenbrock_example.rst, examples/SMAC4HPO/SMAC4HPO_gradientboosting_example.rst, examples/SMAC4HPO/SMAC4HPO_rosenbrock_example.rst, ..., main_options.rst, manual.rst, options.rst, psmac.rst, scenario_options.rst, smac_examples.rst, smac_options.rst, tae.rst, usage_recomendation.rst, validation.rst
generating gallery...
WARNING: /home/skantify/code/SMAC3/examples/quickstart/quickstart_example.py failed to execute correctly: Traceback (most recent call last):                                                    
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/quickstart/quickstart_example.py", line 56, in <module>
    from smac.facade.func_facade import fmin_smac
ModuleNotFoundError: No module named 'smac.facade'


generating gallery for examples/function_minimization... [100%] fmin_rosenbrock_example.py                                                                                                      
WARNING: /home/skantify/code/SMAC3/examples/parallel/fmin_rosenbrock_parallel_example.py failed to execute correctly: Traceback (most recent call last):                                        
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/parallel/fmin_rosenbrock_parallel_example.py", line 12, in <module>
    from smac.intensification.simple_intensifier import SimpleIntensifier
ModuleNotFoundError: No module named 'smac.intensification.simple_intensifier'

WARNING: /home/skantify/code/SMAC3/examples/parallel/parallel_sh_mlp_example.py failed to execute correctly: Traceback (most recent call last):                                                 
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/parallel/parallel_sh_mlp_example.py", line 22, in <module>
    from smac.facade.roar_facade import ROAR
ModuleNotFoundError: No module named 'smac.facade.roar_facade'


generating gallery for examples/SMAC4HPO... [100%] SMAC4HPO_svm_example.py                                                                                                                      
WARNING: /home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_rosenbrock_example.py failed to execute correctly: Traceback (most recent call last):                                               
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_rosenbrock_example.py", line 14, in <module>
    from smac.facade.smac_bo_facade import SMAC4BO
ModuleNotFoundError: No module named 'smac.facade.smac_bo_facade'

WARNING: /home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_sgd_instances_example.py failed to execute correctly: Traceback (most recent call last):                                            
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_sgd_instances_example.py", line 30, in <module>
    from smac.facade.smac_mf_facade import SMAC4MF
ModuleNotFoundError: No module named 'smac.facade.smac_mf_facade'

generating gallery for examples/SMAC4BO... [100%] SMAC4BO_mlp_example.py                                                                                                                        
generating gallery for examples/SMAC4MF... [100%] SMAC4MF_mlp.py                                                                                                                                
WARNING: /home/skantify/code/SMAC3/examples/hyperband/hyperband_mlp_example.py failed to execute correctly: Traceback (most recent call last):                                                  
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/hyperband/hyperband_mlp_example.py", line 25, in <module>
    from smac.facade.hyperband_facade import HB4AC
ModuleNotFoundError: No module named 'smac.facade.hyperband_facade'


computation time summary:
    - ../examples/hyperband/hyperband_mlp_example.py:                 0.01 sec   0.0 MB
    - ../examples/SMAC4BO/SMAC4BO_sgd_instances_example.py:           0.01 sec   0.0 MB
    - ../examples/parallel/parallel_sh_mlp_example.py:                0.00 sec   0.0 MB
    - ../examples/quickstart/quickstart_example.py:                   0.00 sec   0.0 MB
    - ../examples/SMAC4BO/SMAC4BO_rosenbrock_example.py:              0.00 sec   0.0 MB
    - ../examples/parallel/fmin_rosenbrock_parallel_example.py:       0.00 sec   0.0 MB
    - ../examples/SMAC4BO/SMAC4BO_mlp_example.py:                     0.00 sec   0.0 MB
    - ../examples/SMAC4HPO/SMAC4HPO_acq_rosenbrock_example.py:        0.00 sec   0.0 MB
    - ../examples/SMAC4HPO/SMAC4HPO_gradientboosting_example.py:      0.00 sec   0.0 MB
    - ../examples/SMAC4HPO/SMAC4HPO_rosenbrock_example.py:            0.00 sec   0.0 MB
    - ../examples/SMAC4HPO/SMAC4HPO_svm_example.py:                   0.00 sec   0.0 MB
    - ../examples/SMAC4MF/SMAC4MF_mlp.py:                             0.00 sec   0.0 MB
    - ../examples/SMAC4MF/SMAC4MF_sgd_instances.py:                   0.00 sec   0.0 MB
    - ../examples/function_minimization/fmin_rosenbrock_example.py:   0.00 sec   0.0 MB
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 11 source files that are out of date
updating environment: 0 added, 15 changed, 0 removed
reading sources... [100%] smac_options                                                                                                                                                          
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.callbacks'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.configspace'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.epm'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.facade'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.initial_design'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.intensification'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.optimizer'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.runhistory'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.scenario'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.stats'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.tae'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.utils'
/home/skantify/code/SMAC3/doc/api.rst:9: WARNING: toctree contains reference to nonexisting document 'apidoc/smac.smac_cli'
/home/skantify/code/SMAC3/doc/examples/index.rst:22: WARNING: duplicate label sphx_glr_examples_quickstart, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:22: WARNING: duplicate label quickstart, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:59: WARNING: duplicate label sphx_glr_examples_function_minimization, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:59: WARNING: duplicate label fmin_examples, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:94: WARNING: duplicate label sphx_glr_examples_parallel, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:94: WARNING: duplicate label parallel_examples, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:149: WARNING: duplicate label sphx_glr_examples_smac4hpo, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:149: WARNING: duplicate label smac4hpo_examples, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:242: WARNING: duplicate label sphx_glr_examples_smac4bo, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:317: WARNING: duplicate label sphx_glr_examples_smac4mf, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:317: WARNING: duplicate label smac4mf_examples, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:372: WARNING: duplicate label sphx_glr_examples_hyperband, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
/home/skantify/code/SMAC3/doc/examples/index.rst:372: WARNING: duplicate label hyperband_examples, other instance in /home/skantify/code/SMAC3/doc/smac_examples.rst
examples/index.rst:22: WARNING: duplicate label sphx_glr_examples_quickstart, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:22: WARNING: duplicate label quickstart, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:59: WARNING: duplicate label sphx_glr_examples_function_minimization, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:59: WARNING: duplicate label fmin_examples, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:94: WARNING: duplicate label sphx_glr_examples_parallel, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:94: WARNING: duplicate label parallel_examples, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:149: WARNING: duplicate label sphx_glr_examples_smac4hpo, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:149: WARNING: duplicate label smac4hpo_examples, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:242: WARNING: duplicate label sphx_glr_examples_smac4bo, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:317: WARNING: duplicate label sphx_glr_examples_smac4mf, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:317: WARNING: duplicate label smac4mf_examples, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:372: WARNING: duplicate label sphx_glr_examples_hyperband, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
examples/index.rst:372: WARNING: duplicate label hyperband_examples, other instance in /home/skantify/code/SMAC3/doc/examples/index.rst
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] smac_options                                                                                                                                                           
generating indices... genindex py-modindex done
writing additional pages... search done
copying images... [100%] examples/hyperband/images/thumb/sphx_glr_hyperband_mlp_example_thumb.png                                                                                               
copying downloadable files... [100%] examples/quickstart/quickstart_example.ipynb                                                                                                               
copying static files... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 45 warnings.

The HTML pages are in build/html.

Sphinx-gallery successfully executed 0 out of 6 files subselected by:

    gallery_conf["filename_pattern"] = '.*example.py$|.*tutorial.py$'
    gallery_conf["ignore_pattern"]   = '.*_func.py'

after excluding 6 files that had previously been run (based on MD5).


Extension error (sphinx_gallery.gen_gallery):
Handler <function summarize_failing_examples at 0x7f8bea4158b0> for event 'build-finished' threw an exception (exception: Here is a summary of the problems encountered when running the examples

Unexpected failing examples:
/home/skantify/code/SMAC3/examples/quickstart/quickstart_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/quickstart/quickstart_example.py", line 56, in <module>
    from smac.facade.func_facade import fmin_smac
ModuleNotFoundError: No module named 'smac.facade'


/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_sgd_instances_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_sgd_instances_example.py", line 30, in <module>
    from smac.facade.smac_mf_facade import SMAC4MF
ModuleNotFoundError: No module named 'smac.facade.smac_mf_facade'


/home/skantify/code/SMAC3/examples/parallel/parallel_sh_mlp_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/parallel/parallel_sh_mlp_example.py", line 22, in <module>
    from smac.facade.roar_facade import ROAR
ModuleNotFoundError: No module named 'smac.facade.roar_facade'


/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_rosenbrock_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/SMAC4BO/SMAC4BO_rosenbrock_example.py", line 14, in <module>
    from smac.facade.smac_bo_facade import SMAC4BO
ModuleNotFoundError: No module named 'smac.facade.smac_bo_facade'


/home/skantify/code/SMAC3/examples/parallel/fmin_rosenbrock_parallel_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/parallel/fmin_rosenbrock_parallel_example.py", line 12, in <module>
    from smac.intensification.simple_intensifier import SimpleIntensifier
ModuleNotFoundError: No module named 'smac.intensification.simple_intensifier'


/home/skantify/code/SMAC3/examples/hyperband/hyperband_mlp_example.py failed leaving traceback:
Traceback (most recent call last):
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 440, in _memory_usage
    out = func()
  File "/home/skantify/code/SMAC3/.venv/lib/python3.8/site-packages/sphinx_gallery/gen_rst.py", line 425, in __call__
    exec(self.code, self.globals)
  File "/home/skantify/code/SMAC3/examples/hyperband/hyperband_mlp_example.py", line 25, in <module>
    from smac.facade.hyperband_facade import HB4AC
ModuleNotFoundError: No module named 'smac.facade.hyperband_facade'


-------------------------------------------------------------------------------)
make[1]: *** [Makefile:59: html] Error 2
make[1]: Leaving directory '/home/skantify/code/SMAC3/doc'
make: *** [Makefile:12: doc-with-examples] Error 2

opened by eddiebergman 16

numpy runs out of memory

Description

SMAC runs out of memory for some of our scenarios (it has 3.5 GiB available). It catches this and aborts gracefully, but it would be great if there was some way of reducing the amount of memory that SMAC tries to reserve via numpy.

Here is the error traceback:

Traceback (most recent call last):
  File "/infai/seipp/projects/new-benchmarks/optimization/linear.py", line 703, in <module>
    incumbent = smac.optimize()
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/facade/smac_ac_facade.py", line 542, in optimize
    incumbent = self.solver.run()
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/smbo.py", line 201, in run
    challengers = self.choose_next(X, Y)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/smbo.py", line 277, in choose_next
    random_configuration_chooser=self.random_configuration_chooser
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/ei_optimization.py", line 658, in maximize
    _sorted=True,
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/ei_optimization.py", line 554, in _maximize
    return self._sort_configs_by_acq_value(rand_configs)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/ei_optimization.py", line 137, in _sort_configs_by_acq_value
    acq_values = self.acquisition_function(configs)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/acquisition.py", line 77, in __call__
    acq = self._compute(X)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/optimizer/acquisition.py", line 382, in _compute
    m, var_ = self.model.predict_marginalized_over_instances(X)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/epm/rf_with_instances.py", line 269, in predict_marginalized_over_instances
    mean_, var = self.predict(X)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/epm/base_epm.py", line 207, in predict
    mean, var = self._predict(X)
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/smac/epm/rf_with_instances.py", line 223, in _predict
    preds_as_array = np.log(np.nanmean(np.exp(preds_as_array), axis=2) + VERY_SMALL_NUMBER)
  File "<__array_function__ internals>", line 6, in nanmean
  File "/infai/seipp/.conda/envs/smac-conda/lib/python3.7/site-packages/numpy/lib/nanfunctions.py", line 949, in nanmean
    cnt = np.sum(~mask, axis=axis, dtype=np.intp, keepdims=keepdims)
MemoryError: Unable to allocate array with shape (10000, 10, 500) and data type bool

Steps/Code to Reproduce

I don't have a minimal example to reproduce the error, but here are our logs and SMAC output files: https://ai.dmi.unibas.ch/tmp_files/seipp/smac-numpy-out-of-memory.tar.gz You can find the stdout and stderr output in run.log and run.err. The smac files are under smac/run*.

Do you have any suggestion how to reduce the memory usage?

Versions

0.11.1

documentation

opened by jendrikseipp 16

How to generate same challengers in every runs?

As title, I wanna get the same result in different runs, how can I do? does it can generate same challengers in every runs but different in each iterations?

opened by qqliang 16
Fix#55
Hi,

restructured the SMAC code; the main changes is that I added facades to abstract away some complexity. Right now, I have two facades

standard SMAC over command line

a scipy similar facade (see examples/rosenbrock2.py for an example to use this new facade)

@mfeurer and @KEggensperger : could you two please have a look at the changes and whether you agree on them.

I'm aware that a lot of unit tests are probably broken right now. I would like to have first some feedback on the changes before I start to fix the unit tests.

Best, Marius
discussion
opened by mlindauer 16
ADD initial design with multiple configuration read from previous
ADD methods to read trajectories from disc (in aclib2 json format)

ADD initial design based on multiple configurations using intensification to determine initial incumbent

ADD adaption of SMAC facade to support the new initial design

FIX adaptive capping slack use -- I know this should not be in this PR, sorry!

MAINT move check "first run crashed" from initial design into TAE

FIX maximal wc limit by pynisher to (2**16)-1

FIX single config initial design adds also the run to the runhistory

MAINT argument to disable traj logging

@mfeurer @KEggensperger Could you please review this PR. If you approve the changes, I will add further unit tests.
opened by mlindauer 15

getting final result from pSMAC on a distributed compute cluster

Hi,

I have managed to run pSMAC on a distributed DASK cluster. I run it as follows:

cs = ConfigurationSpace()
scenario = Scenario({"run_obj": "quality",  
                         "runcount-limit": 50,  
                         "cs": cs,              
                         "deterministic": True,
                         "shared_model": True, "initial_incumbent": 'RANDOM',
                         "output_dir": "/shared_data/smac3-output/runs/",
                         "input_psmac_dirs": "/shared_data/smac3-output/runs/run_*"})

def errortomimize:
  return( trainmodel( **cfg), use_pynisher = False))
tae = ExecuteTAFuncDict(errortomimize,use_pynisher=False)
smac = SMAC(scenario=scenario, rng=np.random.RandomState(i), tae_runner=tae, run_id = i)

where i in this case is the worker number.

This creates a directory structure in the shared space like the following:

----smac3-output/runs
                                |
                                |_______run_1
                                |_______run_2
                                             . . .
                                |_______run_N

where N is the number of workers in the cluster.

However, if you look at the traj_aclib2.json in each run_* subdirectory, they are identical. I tried adding verbose_level = "DEBUG" to errortomimize but I don't get any output from this.

Any idea what the problem is? How can I tell whether pSMAC is actually sharing the model (as opposed to just running the same thing N times)?

Cheers, Noah

stale

opened by Mestalbet 14

Version 2.0.0
Required for RC

[ ] Integrate BOinG + TurBO

[ ] Integrate HydraFacade

[ ] Bug when evaluating only few trials?

[ ] Run profiler

[ ] Use latest budget if highest budget not yet evaluated (or give option)

[ ] Option to restart crashed ones

Discussions

Continue runs.

Many components are depending on states. Idea: Save/load states s.t. the optimization can pick-off where it stopped.

Have the option to re-run crashed ones.

HyperparameterFacade starts a local search when continuing a run. However, the limit already was exceeded.

HB/SH does not work because an incumbent is given and it's stuck in the first stage.

Ask-and-Tell Interface

Call asks multiple times before calling tell

Find a way to incorporate the trials from the user when only using tell. Partially works for intensifier already.

It does not make sense to tell SMAC trials in advance when using SH. Reason: It's heavily depending on a budget+instance combination and even if the user provides it, SMAC have to wait till the other trials have been finished too.

Constraints instead of imputation?

Facade: Build the facade automatically based on the scenario inputs (like if budgets are defined use successivehalving e.g.) -> AutoFacade? -> Log which components are used

Random_design.check should not be in acquition but in SMBO main loop, since it obleviates the necessity of computing epm and acquition to begin with?

Ask and intensification inversion. Currently, the ask method is passed to the intensifier. But ideally, the ask method defines what configuration to be evaluated exactly on what problem instance. This implies the inverse relation: the intensifier should be called from within ask?

Intensification is one way of defining a fidelity (as number of problem instances to evaluate on) but it shouldn’t be at the heart of SMAC, since nowadays the multiple dataset optimization is no longer as prominent.

10000 challengers: Never touch the surrogate model again? This usually happens due to the intensification percentage being at 0.5, the model fitting taking quite long, and the functions taking no time. If these three things don't apply, this is indeed an issue

_collect_data in smbo.py: Training only on the highest fidelity or mixed fidelity? -> Docs

Problem with _get_x_best and instances: Only the config with the lowest cost is used?!

Tools to visualize things.
opened by renesass 0
Limitation in number of Hyperparameters with Multi-Fidelity

question

Description

Hi, I would like to use Multi-Fidelity option of SMAC to find optimum arrangement defined by about 2752 Hyperparameters within ConfigurationSpace.

I was wondering if there is any performance limitation in number of hyperparameters for Optimization by using your implemetation? these 2752 Hyperparameters can be all categorial or all integers.

Thanks for your support and best regards

opened by Mshz2 2
Algorithm configuration memory issue
Hi,

The current version of SMAC can not run algorithm configuration scenarios that where working with version 0.10. In particular, it seems that from version 0.11 there are memory problems: some scenarios crash after about 20 minutes because they reach dozens of Gb of memory (could be linked to issue 187 ).

You can find a zip file with a few log files and the details of the scenarios on which SMAC was run to obtain them here

Note that: I am running SMAC from command line; The results I am sending you where all run from the same conda environment so they use the same version of the libraries; For the logs that end with "CANCELLED" it is simply that I stopped them after a bit more than 24 hours, logs of seed 5 and 6 are in debug mode.

A high level look at the log file:

the memory problem seems to appear from version 0.11 (see logs of scenario sparrowtoriss on circuitfuzz)

I have had memory problems with SMAC3-v.1.4 on scenarios using Sparrowtoriss, Clasp, Lingeling, but no problem with cplex and kissat (pcs file in the archive)

the number of steps of the local search seems to go from more than 10 for each LS in v0.10 to about 2 in v0.11, which could mean that the local search is not much better than random search from then. (see debug output of kissat on circuitfuzz) However, it might also be that the debug output is giving numbers that are not meaningful. When looking at the code we found that the time per acquisition function "vectorized" is an average over non comparable values.

the reported "Average configurations per intensify" goes from ~14 in v0.10 to ~30 in v0.11 and then drops to 0 from v1.0

bug
opened by MarieNyst 6
[Q] Different instances across different runs

Is it possible to incrementally (i.e., over multiple runs) add datasets and have SMAC3 find the optimum on the current dataset taking into account trial results from previous optimization sessions on older instances?

If, e.g., every month one gets another dataset which is more correlated to the dataset from last month than to the dataset from two months ago, is it possible to do a monthly search for the optimum on the current data that also takes into account the results of the trials executed over the sessions in the previous months?

I think this may be somewhat related to #804, but I'm a bit confused by the docs, as there is no clear example of this. The v2.0.0a2 docs (here's the current version of the doc source) is a bit contradictory or unclear, since first it states that "SMAC automatically restores states", but then that "[...] it is not possible to continue a run from a previous state yet". Ok, this is an alpha version so this stuff is kind of expected.

On the other hand, in the stable v1.4.0 docs, there is the restore branin example, which shows that old history can be incorporated into new scenarios, but the page on Restoring warns that modifying parameters other than output_dir, wallclock_limit, runcount_limit or tuner-timeout "will likely lead to unexpected behaviour" (this comment that provides a solution also warns for side effects).

Or maybe I'm asking the question wrong. I was thinking of instances, but that might not be the optimal way to get what I need. Maybe all that's required is to just constrain the search space to a sub-space of one less dimension corresponding to a single value of one the parameters (like Optuna's FixedPartialSampler), but couldn't find any docs for that either. Then again, the config space will change between sessions, and I understand this might be problematic, right?

opened by bbudescu 1

Hydra facade not working

Description

I am trying to use the hydra facade in smac/facade/experimental, however, it gives an error.

Steps/Code to Reproduce

def dnnv_from_cfg(cfg, seed, instance):

    #TAE function

    return score

if __name__ == "__main__":

    cs = ConfigurationSpace({

        })

    scenario = Scenario(
        {
            "run_obj": "quality", 
            "cutoff_time": 300,
            "ta_run_limit": 300,
            "wallclock_limit": 3600,
            "cs": cs,
            "deterministic": True,
            "train_inst_fn": 'instances.txt'
        }
    )

    # smac = SMAC4AC(
    #     scenario=scenario,
    #     tae_runner=dnnv_from_cfg
    # )

    # incumbent = smac.optimize()

    hydra = Hydra(scenario=scenario, n_iterations=2, tae=dnnv_from_cfg)
    hydra.optimize()

Expected Results

Hydra starts the configuration procedure.

Actual Results

INFO:smac.utils.io.cmd_reader.CMDReader:Output to smac3-output_2022-11-09_11:39:05_493000
Optimizing! Depending on your machine, this might take a few minutes.
INFO:smac.facade.smac_ac_facade.SMAC4AC:Optimizing a deterministic scenario for quality without a tuner timeout - will make SMAC deterministic and only evaluate one configuration per iteration!
INFO:smac.scenario.scenario.Scenario:No output directory for scenario logging specified -- scenario will not be logged.
INFO:smac.utils.io.traj_logging.TrajLogger:No output directory for trajectory logging specified -- trajectory will not be logged.
INFO:smac.initial_design.default_configuration_design.DefaultConfiguration:Running initial design for 1 configurations
INFO:smac.facade.experimental.hydra_facade.Hydra:========================================================================================================================
INFO:smac.facade.experimental.hydra_facade.Hydra:Hydra Iteration: 1
INFO:smac.facade.psmac_facade.PSMAC:++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
INFO:smac.facade.psmac_facade.PSMAC:PSMAC run
Traceback (most recent call last):
  File "run_dnnv.py", line 104, in <module>
    hydra.optimize()
  File "/.dnnv-env/lib/python3.7/site-packages/smac/facade/experimental/hydra_facade.py", line 214, in optimize
    incs = self.optimizer.optimize()
  File "/.dnnv-env/lib/python3.7/site-packages/smac/facade/psmac_facade.py", line 218, in optimize
    for seed in self.seeds
  File "/.dnnv-env/lib/python3.7/site-packages/joblib/parallel.py", line 1098, in __call__
    self.retrieve()
  File "/.dnnv-env/lib/python3.7/site-packages/joblib/parallel.py", line 975, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/.dnnv-env/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 567, in wrap_future_result
    return future.result(timeout=timeout)
  File "/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
TypeError: __init__() got an unexpected keyword argument 'tae'

Versions

smac 1.4.0

opened by marti-mcfly 3

Feature request: Optimizing hyperparameters under unknown constraints
Description

The ability to optimize along with a univariate hyperparameter an uknown/hidden constraint for the parameter, where invalid parameter values are identified by failed function evaluations.

This may be related to #403.

Specific application

For the simplest case I have, we can assume:

the hyperparameter $x$ is independent of all other hyperparameters

all valid values occur below some threshold $x_c$. All $x \ge x_c$ are invalid.

so once an invalid value is encountered, we never want to test a value larger than it.

An initial idea

The KI Campus MOOC on AutoML mentioned that the Expected Constrained Improvement acquisition function, given in Eq.4 of https://engineering.ucsc.edu/sites/default/files/technical-reports/UCSC-SOE-10-10.pdf as $$x' = \arg\max_{x \in \mathcal{X}} \mathbb{E}[I(x)] h(x)$$ is one approach that could work for something like this, where $I$ is an improvement statistic, and $h(x)$ is the predicted probability that the parameter $x$ is valid, for which they use a random forest classifier.

I suppose a more general approach would allow a user to provide

a function $h: \mathcal{X} \to [0, 1]$, which could be multiplied by any acquisition function

a method to update $h$ given a new evaluation $x_k$,

feature
opened by sethaxen 2

Releases(v2.0.0-beta.1)

v2.0.0-beta.1(Jan 7, 2023)
Completely reimplemented the intensifiers (including Successive Halving and Hyperband): All intensifiers support multi-fidelity, multi-objective and multi-threading by nature now.

Expected behaviour for ask-and-tell interface ensured (also for Successive Halving).

Continuing a run is now fully supported.

Added more examples.

Updated documentation based on new implementation.

Added benchmark to compare different versions.

Source code(tar.gz)
Source code(zip)
v2.0.0-alpha.2(Oct 26, 2022)
Bugfixes

Fixed random weight (re-)generalization of multi-objective algorithms: Before the weights were generated for each call to build_matrix, now we only re-generate them for every iteration.

Optimization may get stuck because of deep copying an iterator for callback: We removed the configuration call from on_next_configurations_end.

Minor

Removed example badget in README.

Added SMAC logo to README.

Source code(tar.gz)
Source code(zip)
v2.0.0-alpha.1(Oct 12, 2022)
Big Changes

We redesigned the scenario class completely. The scenario is implemented as a dataclass now and holds only environment variables (like limitations or save directory). Everything else was moved to the components directly.

We removed runtime optimization completely (no adaptive capping or imputing anymore).

We removed the command-line interface and restructured everything alongside. Since SMAC was building upon the command-line interface (especially in combination with the scenario), it was complicated to understand the behavior or find specific implementations. With the removal, we re-wrote everything in python and re-implemented the feature of using scripts as target functions.

Introducing trials: Each config/seed/budget/instance calculation is a trial.

The configuration chooser is integrated into the SMBO object now. Therefore, SMBO finally implements an ask-tell interface now.

Facades are redesigned so that they accept instantiated components directly. If a component is not passed, a default component is used, which is specified for each facade individually in the form of static methods. You can use those static methods directly to adapt a component to your choice.

A lot of API changes and renamings (e.g., RandomConfigurationChooser -> RandomDesign, Runhistory2EPM -> RunHistoryEncoder).

Ambiguous variables are renamed and unified across files.

Dependencies of modules are reduced drastically.

We incorporated Pynisher 1.0, which ensures limitations cross-platform.

We incorporated ConfigSpace 0.6, which simplified our examples.

Examples and documentation are completely reworked. Examples use the new ConfigSpace, and the documentation is adapted to version 2.0.

Transparent target function signatures: SMAC checks now explicitly if an argument is available (the required arguments are now specified in the intensifier). If there are more arguments that are not passed by SMAC, a warning is raised.

Components implement a meta property now, all of which describe the initial state of SMAC. The facade collects all metadata and saves the initial state of the scenario.

Improved multi-objective in general: RunHistory (in addition to RunHistoryEncoder) both incorporates the multi-objective algorithm. In other words, if the multi-objective algorithm changes the output, it directly affects the optimization process.

Configspace is saved in json only

StatusType is saved as integer and not as dict anymore

We changed the behavior of continuing a run:

SMAC automatically checks if a scenario was saved earlier. If there exists a scenario and the initial state is the same, SMAC automatically loads the previous data. However, continuing from that run is not possible yet.

If there was a scenario earlier, but the initial state is different, then the user is asked to overwrite the run or to still continue the run although the state is different (Note that this only can happen if the name specified in the scenario is the same). Alternatively, an old to the old run is added (e.g., the name was test, it becomes test-old).

The initial state of the SMAC run also specifies the name (if no name in the scenario is specified). If the user changes something in the code base or in the scenario, the name and, therefore, the save location automatically changes.

New Features

Added a new termination feature: Use terminate_cost_threshold in the scenario to stop the optimization after a configuration was evaluated with a cost lower than the threshold.

Callbacks are completely redesigned. Added callbacks to the facade are called in different positions in the Bayesian optimization loop.

The multi-objective algorithm MeanAggregationStrategy supports objective weights now.

RunHistory got more methods like get_incumbent or get_pareto_front.

Fixes

You ever noticed that the third configuration has no origin? It's fixed now.

We fixed ParEGO (it updates every time training is performed now).

Optimization Changes

Changed initial design behavior

You can add additional configurations now.

max_ratio will limit both n_configs and n_configs_per_hyperparameter but not additional configurations

Reduced default max_ratio to 0.1.

Code Related

Converted all unittests to pytests.

Instances, seeds, and budgets can be set to none now. However, mixing none and non-none will throw an exception.

Source code(tar.gz)
Source code(zip)
v1.4.0(Jul 14, 2022)
Features

BOinG: A two-stage Bayesian optimization approach to allow the optimizer to focus on the most promising regions.

TurBO: Reimplementaion of TurBO-1 algorithm.

Updated pSMAC: Can pass arbitrary SMAC facades now. Added example and fixed tests.

Improvements

Enabled caching for multi-objectives (#872). Costs are now normalized in get_cost or optionally in average_cost/sum_cost/min_cost to receive a single float value. Therefore, the cached cost values do not need to be updated everytime a new entry to the runhistory was added.

Interface changes

We changed the location of Gaussian processes and random forests. They are in the folders epm/gaussian_process and epm/random_forest now.

Also, we restructured the optimizer folder and therefore the location of the acquisition functions and configuration chooser.

Multi-objective functions are located in the folder multi_objective.

pSMAC facade was moved to the facade directory.

Source code(tar.gz)
Source code(zip)
v1.3.4(Jun 23, 2022)
Added reference to JMLR paper.

Typos in documentations.

Code more readable since all typings are imported at the beginning of the file.

Updated stale bot options.

Source code(tar.gz)
Source code(zip)
v1.3.3(May 19, 2022)
Hotfix: Since multi-objective implementation depends on normalized costs, it now is ensured that the cached costs are updated everytime a new entry is added.

Removed mac-specific files.

Added entry point for cli.

Added ConfigSpace to third known parties s.t. sorting should be the same across different operating systems.

Fixed bugs in makefile in which tools were specified incorrectly.

Executed isort/black on examples and tests.

Updated README.

Fixed a problem, which incremented time twice before taking log (#833).

New wrapper for multi-objective models (base_uncorrelated_mo_model). Makes it easier for developing new multi-objective models.

Raise error if acquisition function is incompatible with the epm models.

Restricting pynisher.

Source code(tar.gz)
Source code(zip)
v1.3.2(May 5, 2022)
Added stale bot support.

If package version 0.0.0 via get_distribution is found, the version of the module is used instead.

Removed tox.ini.

Moved requirements.txt to setup.py.

Added multi-objective support for ROAR.

Added notes in documentation that SMAC4MF is the closest implementation to BOHB/HpBandSter.

Source code(tar.gz)
Source code(zip)
v1.3.1(Apr 22, 2022)
Added Python 3.7 support again.

Source code(tar.gz)
Source code(zip)
v1.3.0(Apr 19, 2022)
Features

PiBO: Augment the acquisition function by multiplying by a pdf given by the user. The prior then decays over time, allowing for the optimization to carry on as per default.

The RunHistory can now act as a Mapping in that you can use the usual methods you can use on dicts, i.e. len(rh), rh.items(), rh[key]. Previously this was usually done by accessing rh.data which is still possible.

Minor Changes

Updated the signature of the ROAR facade to match with it's parent class SMAC4AC. Anyone relying on the output directory without specifying an explicit run_id to a ROAR facade should now expect to see the output directory at run_0 instead of run_1. See #827.

Code-Quality

Updated and integrated flake8, mypy, black, and isort.

Documentation

SMAC uses automl_sphinx_theme now and therefore the API is displayed nicer.

Source code(tar.gz)
Source code(zip)
v1.2.0(Feb 17, 2022)
Features

Added multi-objective optimization via Mean-Aggregation or Par-EGO (#817, #818). Both approaches normalize the costs objective-wise based on all data in the history.

Major Changes

Results are instantly saved by default now. That means, runhistory.json is saved every time a trial is added.

Determinstic behaviour (defined in scenario) is default now. Calling a function/TAE with the same seed and configuration is expected to be the same.

Limit resources behaviour is by default false now. This is particually important because pynisher does not work on all machines (e.g. Colab, Mac, Windows, ...) properly.

Renamed scenario object save_results_instantly to save_instantly.

Added multi_objectives as scenario argument.

Expanded cost_for_crash for multi-objective support.

Examples

Integrated spear_qcp example for commandline.

Python examples are now executed so that the output in the documentation is shown.

Added multi-objective example.

Documentation

Added runhistory page.

Workflow Clean-up

Adds PEP 561 compliance (exports types so other packages can be aware of them).

Allow manual workflow_dispatch on actions that might require it (can manually trigger them from github UI).

Prevent the double trigger of actions by making push and pull_request and more strict.

A push to a pull request should no longer cause double the amount of tests to run (along with the other workflows that had on: [push, pull_request].

Some general cleanup, giving names to some actions, adding some linebreaks to break up things, ...

Command-line examples are tested again.

pytest.yaml:

Now scheduled to auto run everyday instead of every week.

Clean up the body of the steps and move some things to env var.

Scaffold for matrix that includes windows and mac testing (currently excluded, see comments).

Includes tests for Python 3.10.

Changed the boolean flags in the matrix to just be a categorical, easier to read.

Minor Changes

Specified that dask should not cache functions/results (#803) .

Handles invalid configuration vectors gracefully (#776).

Specified scenario docs that also SMAC options can be used.

Docs display init methods now.

Parameters in the docs are shown first now.

Successive Halving only warns you once if one worker is used only.

Statistics are better readable now.

Sobol sequence does not print warnings anymore.

Source code(tar.gz)
Source code(zip)
v1.1.1(Jan 27, 2022)
Minor Changes

Added comparison between SMAC and similar tools.

Updated installation guide.

Added a warning that CLI is only available when installing from GitHub.

Source code(tar.gz)
Source code(zip)
v1.1.0(Oct 21, 2021)
Major Changes

Option to use an own stopping strategy using IncorporateRunResultCallback.

Documentation was updated thoroughly. A new theme with a new structure is provided and all pages have been updated. Also, the examples revised and up-to-date.

Changed scripts/smac to scripts/smac.py.

Minor Changes

README.md updated.

CITATION.cff added.

Made smac-validate.py consistent with runhistory and tae. (#762)

minR, maxR and use_ta_time can now be initialized by the scenario. (#775)

ConfigSpace.util.get_one_exchange_neighborhood's invalid configurations are ignored. (#773)

Bug Fixes

Fixed an incorrect adaptive capping behaviour. (#749)

Avoid the potential ValueError raised by LocalSearch._do_search. (#773)

Source code(tar.gz)
Source code(zip)
v1.0.1(Aug 6, 2021)
1.0.1

Minor Changes

Added license information to every file.

Fixed a display bug inside usage recommendation.

Source code(tar.gz)
Source code(zip)
v1.0.0(Aug 6, 2021)
1.0.0

The main purpose of this release is to be synchronized with our upcoming paper. Since many urgent features were already taken care of in 0.14.0, this release mainly focuses on better documentation and examples.

Features

Examples and quickstart guide can now be generated by sphinx-gallry.

Added make command make doc and make doc-with-examples.

Major changes

Examples are separated into categories.

Renamed facade SMAC4BO to SMAC4BB (black-box).

Add thompson sampling as a new acquisition function.

Minor Changes

Included linkcheck and buildapi to the make doc command.

quickstart.rst was converted to quickstart_example.py to be processed by sphinx-gallery.

Examples renamed from *.py to *_example.py, unless file name was *_func.py, in which case it was unchanged.

Flake8 fixes for spear_qcp as there were a lot of complaints running pre-commit.

Fixes pydoc issues.

Fixed links in the README.

Fixed warnings given during the doc build.

Fixed inconsistent output shape described in smac.epm.gaussian_process.GaussianProcess.sample_functions

Examples are wrapped inside if __name__ == "__main__", fixing problems on mac.

Source code(tar.gz)
Source code(zip)
v0.14.0(Aug 5, 2021)
Breaking Changes

BOHB4HPO facade has been renamed to SMAC4MF facade (#738)

Require scipy >= 1.7 (#729)

Require emcee >= 3.0.0 (#723)

Major Changes

Drop support for Python 3.6 (#726)

Added Colab to try SMAC in your browser! (#697)

Minor Changes

Added gradient boosting example, removed random forest example (#722)

lazy_import dependency dropped (#741)

Replaced pyDOE requirement with scipy for LHD design (#735)

Uses scrambled Sobol Sequence (#733)

Moved to Github actions (#715)

Improved testing (#720, #723, #739, #743)

Added option save_results_instantly in scenario object to save results instantly (#728)

Changed level of intensification messages to debug (#724)

Bug Fixes

Github badges updated (#732)

Fixed memory limit issue for pynisher (#717)

More robust multiprocessing (#709, #712)

Fixed serialization with runhistory entries (#706)

Separated evaluation from get next challengers in intensification (#734)

Doc fixes (#727, #714)

Source code(tar.gz)
Source code(zip)
v0.13.1(Mar 15, 2021)
0.13.1

Minor Changes

Improve error message for first run crashed (#694).

Experimental: add callback mechanism (#703).

Bug fixes

Fix a bug which could make successive halving fail if run in parallel (#695).

Fix a bug which could cause hyperband to ignore the lowest budget (#701).

Source code(tar.gz)
Source code(zip)
v0.13.0(Sep 25, 2020)
Major Changes

Split choosing next challenger from evaluating challenger (#663)

Implemented parallel SMAC using dask (#675, #677, #681, #685, #686)

Drop support for Python 3.5

Minor Changes

Update Readme

Remove runhistory from TAE (#663)

Store SMAC's internal config id in the configuration object (#679)

Introduce Status Type STOP (#690)

Bug Fixes

Only validate restriction of Sobol Sequence when choosing Sobol Sequence (#664)

Fix wrong initialization of list in local search (#680)

Fix setting random seed with a too small range in Latin Hypercube design (#688)

Source code(tar.gz)
Source code(zip)
v0.12.3(Jul 31, 2020)
Release 0.12.3

Minor Changes

Use Scipy's Sobol sequence for the initial design instead of a 3rd-party package (#600)

Store start and end time of function evaluation (#647)

Bug Fixes

Fixes an issue in the Bayesian optimization facade which triggered an exception when tuning categorical hyperparameters (#666)

Fixes an issue in the Gaussian process MCMC which resulted in reduced execution speed and reduced performance (#666)

Source code(tar.gz)
Source code(zip)
v0.12.2(May 25, 2020)
Bug Fixes

Fixes the docstring of SMAC's default acquisition function optimizer (#653)

Correctly attributes the configurations' origin if using the FixedSet acquisition function optimizer (#653)

Fixes an infinite loop which could occur if using only a single configuration per iteration (#654)

Fixes a bug in the kernel construction of the BOFacade (#655)

Source code(tar.gz)
Source code(zip)
v0.12.1(May 6, 2020)
Minor Changes

Upgrade the minimal scikit-learn dependency to 0.22.X.

Make GP predictions faster (#638)

Allow passing tae_runner_kwargs to ROAR.

Add a new StatusType DONOTADVANCE for runs that would not benefit from a higher budgets. Such runs are always used to build a model for SH/HB (#632)

Add facades/examples for HB/SH (#610)

Compute acquisition function only if necessary (#627,#629)

Bug Fixes

Fixes a bug which caused SH/HB to consider TIMEOUTS on all budgets for model building (#632)

Fixed a bug in adaptive capping for SH (#619,#622)

Source code(tar.gz)
Source code(zip)
v0.12.0(Mar 2, 2020)
Major Changes

Support for Successive Halving and Hyperband as new instensification/racing strategies.

Improve the SMAC architecture by moving from an architecture where new candidates are passed to the racing algorithm to an architecture where the racing algorithm requests new candidates, which is necessary to implement the BOHB algorithm (#551).

Source code is now PEP8 compliant. PEP8 compliance is checked by travis-ci (#565).

Source code is now annotated with type annotation and checked with mypy.

Minor Changes

New argument to directly control the size of the initial design (#553).

Acquisition function is fed additional arguments at update time (#557).

Adds new acquisition function maximizer which goes through a list of pre-specified configurations (#558).

Document that the dependency pyrfr does not work with SWIG 4.X (#599).

Improved error message for objects which cannot be serialized to json (#453).

Dropped the random forest with HPO surrogate which was added in 0.9.

Dropped the EPILS facade which was added in 0.6.

Simplified the interface for constructing a runhistory object.

removed the default rng from the Gaussian process priors (#554).

Adds the possibility to specify the acquisition function optimizer for the random search (ROAR) facade (#563).

Bump minimal version of ConfigSpace requirement to 0.4.9 (#578).

Examples are now rendered on the website using sphinx gallery (#567).

Bug fixes

Fixes a bug which caused SMAC to fail for Python function if use_pynisher=False and an exception was raised (#437).

Fixes a bug in which samples from a Gaussian process were shaped differently based on the number of dimesions of the y-array used for fitting the GP (#556).

Fixes a bug with respect saving data as json (#555).

Better error message for a sobol initial design of size >40 ( #564).

Add a missing return statement to GaussianProcess._train.

Source code(tar.gz)
Source code(zip)
v0.11.1(Sep 20, 2019)
Updated the default hyperparameters of the Gaussian process facade to follow recent research (#529)

Enabled flake8 code style checks for newly merged code (#525)

Source code(tar.gz)
Source code(zip)
v0.11.0(Aug 29, 2019)
Major changes

Local search now starts from observed configurations with high acquisition function values, low cost and the from unobserved configurations with high acquisition function values found by random search (#509)

Reduces the number of mandatory requirements (#516)

Make Gaussian processes more resilient to linalg error by more aggressively adding noise to the diagonal (#511)

Inactive hyperparameters are now imputed with a value outside of the modeled range (-1) (#508)

Replace the GP library George by scikit-learn (#505)

Renames facades to better reflect their use cases (#492), and adds a table to help deciding which facade to use (#495)

SMAC facades now accept class arguments instead of object arguments (#486)

Minor changes

Vectorize local search for improved speed (#500)

Extend the Sobol and LHD initial design to work for non-continuous hyperparameters as well applying an idea similar to inverse transform sampling (#494)

Bug fixes

Fixes a regression in the validation scripts (#519)

Fixes a unit test regression with numpy 1.17 (#523)

Fixes an error message (#510)

Fixes an error making random search behave identical for all seeds

Source code(tar.gz)
Source code(zip)
v0.10.0(Jan 24, 2019)
Major changes

ADD further acquisition functions: PI and LCB *SMAC can now be installed without installing all its dependencies

Simplify setup.py by moving most thing to setup.cfg

Bug fixes

RM typing as requirement

FIX import of authors in setup.py

MAINT use json-file as standard pcs format for internal logging

Source code(tar.gz)
Source code(zip)
v0.9.0(Nov 16, 2018)
Major changes

ADD multiple optional initial designs: LHC, Factorial Design, Sobol

ADD fmin interface know uses BORF facade (should perform much better on continuous, low-dimensional functions)

ADD Hydra (see "Hydra: Automatically Configuring Algorithms for Portfolio-Based Selection" by Xu et al)

MAINT Not every second configuration is randomly drawn, but SMAC samples configurations randomly with a given probability (default: 0.5)

MAINT parsing of options

Interface changes

ADD two new interfaces to optimize low dimensional continuous functions (w/o instances, docs missing)

BORF facade: Initial design + Tuned RF

BOGP interface: Initial design + GP

ADD options to control acquisition function optimization

ADD option to transform function values (log, inverse w/ and w/o scaling)

ADD option to set initial design

Minor changes

ADD output of estimated cost of final incumbent

ADD explanation of "deterministic" option in documentation

ADD save configspace as json

ADD random forest with automated HPO (not activated by default)

ADD optional linear cooldown for interleaving random configurations (not active by default)

MAINT Maximal cutoff time of pynisher set to UINT16

MAINT make SMAC deterministic if function is deterministic, the budget is limited and the run objective is quality

MAINT SLS on acquisition function (plateau walks)

MAINT README

FIX abort-on-first-run-crash

FIX pSMAC input directory parsing

FIX fmin interface with more than 10 parameters

FIX no output directory if set to '' (empty string)

FIX use np.log instead of np.log10

FIX No longer use law of total variance for uncertainty prediction for RFs as EPM, but only variance over trees (no variance in trees)

FIX Marginalize over instances inside of each tree of the forest leads to better uncertainty estimates (motivated by the original SMAC implementation)

Source code(tar.gz)
Source code(zip)
v0.8.0(Dec 15, 2017)
Major changes

Upgrade to ConfigSpace (0.4.X), which is not backwards compatible. On the plus side, the ConfigSpace is about 3-10 times faster, depending on the task.

FIX #240: improved output directory structure. If the user does not specify an output directory a SMAC experiment will have the following structure: smac_/run_<run_id>/*.json. The user can specify a output directory, e.g. ./myExperiment or ./myExperiment/ which results in ./myExperiment/run_<run_id>/*.json.

Due to changes in AnaConda's compiler setup we drop the unit tests for python3.4.

Interface changes

Generalize the interface of the acquisition functions to work with ConfigSpaces's configuration objects instead of numpy arrays.

The acquisition function optimizer can now be passed to the SMBO object.

A custom SMBO class can now be passed to the SMAC builder object.

run_id is no longer an argument to the Scenario object, making the interface a bit cleaner.

Minor changes

#333 fixes an incompability with uncorrelated_mo_rf_with_instances.

#323 fixes #324 and #319, which both improve the functioning of the built-in validation tools.

#350 fixes random search, which could accidentaly use configurations found my a local acquisition function optimizer.

#336 makes validation more flexible.

Source code(tar.gz)
Source code(zip)
v0.7.1(Oct 12, 2017)

FIX #193, restoring the scenario now possible ADD #271 validation FIX #311 abort on first crash FIX #318, ExecuteTARunOld now always returns a StatusType
Source code(tar.gz)
Source code(zip)
v0.6.0(Aug 7, 2017)

Major changes

MAINT documentation (nearly every part was improved and extended, including installation, examples, API) ADD EPILS as mode (modified version of ParamILS) MAINT minimal required versions of configspace, pyrfr, sklearn increased (several issues fixed in new configspace version) MAINT for quality scenarios, the user can specify the objective value for crashed runs (returned NaN and Inf are replaced by value for crashed runs)

Minor changes

FIX issue #220, do not store external data in runhistory MAINT TAEFunc without pynisher possible MAINT intensification: minimal number of required challengers parameterized FIX saving duplicated (capped) runs FIX handling of ordinal parameters MAINT runobj is now mandatory FIX arguments passed to pyrfr
Source code(tar.gz)
Source code(zip)
v0.5.0(May 8, 2017)
Major changes

MAINT #192: SMAC uses version 0.4 of the random forest library pyrfr. As a side-effect, the library swig is necessary to build the random forest.

MAINT: random samples which are interleaved in the list of challengers are now obtained from a generator. This reduces the overhead of sampling random configurations.

FIX #117: only round the cutoff when running a python function as the target algorithm.

MAINT #231: Rename the submodule smac.smbo to smac.optimizer.

MAINT #213: Use log(EI) as default acquisition function when optimizing running time of an algorithm.

MAINT #223: updated example of optimizing a random forest with SMAC.

MAINT #221: refactored the EPM module. The PCA on instance features is now part of fitting the EPM instead of reading a scenario. Because of this restructuring, the PCA can now take instance features which are external data into account.

Minor changes

SMAC now outputs scenario options if the log level is DEBUG (2f0ceee).

SMAC logs the command line call if invoked from the command line (3accfc2).

SMAC explicitly checks that it runs in python>=3.4.

MAINT #226: improve efficientcy when loading the runhistory from a json file.

FIX #217: adds milliseconds to the output directory names to avoid race. conditions when starting multiple runs on a cluster.

MAINT #209: adds the seed or a pseudo-seed to the output directory name for better identifiability of the output directories.

FIX #216: replace broken call to in EIPS acqusition function.

MAINT: use codecov.io instead of coveralls.io.

MAINT: increase minimal required version of the ConfigSpace package to 0.3.2.

Source code(tar.gz)
Source code(zip)
0.4.0(Apr 4, 2017)
ADD #204: SMAC now always saves runhistory files as runhistory.json.

MAINT #205: the SMAC repository now uses codecov.io instead of coveralls.io.

ADD #83: support of ACLIB 2.0 parameter configuration space file.

FIX #206: instances are now explicitly cast to str. In case no instance is given, a single None is used, which is not cast to str.

ADD #200: new convenience function to retrieve an X, y representation of the data to feed it to a new fANOVA implementation.

MAINT #198: improved pSMAC interface.

FIX #201: improved handling of boolean arguments to SMAC.

FIX #194: fixes adaptive capping with re-occurring configurations.

ADD #190: new argument intensification_percentage.

ADD #187: better dependency injection into main SMAC class to avoid ill-configured SMAC objects.

ADD #161: log scenario object as a file.

ADD #186: show the difference between old and new incumbent in case of an incumbent change.

MAINT #159: consistent naming of loggers.

ADD #128: new helper method to get the target algorithm evaluator.

FIX #165: default value for par = 1.

MAINT #153: entries in the trajectory logger are now named tuples.

FIX #155: better handling of SMAC shutdown and crashes if the first configuration crashes.

Source code(tar.gz)
Source code(zip)

Sequential Model-based Algorithm Configuration

Related tags

Overview

SMAC v3 Project

OVERVIEW

Installation

Requirements

Installation via pip

Manual Installation

Installation in Anaconda

Optional dependencies

License

USAGE

Examples

Contact

Comments

PiBO - BO with user Priors over the Optimum

New attempt on PiBO - here are the changes from last time:

New Added unit tests for PriorAcquisitionFunction (and required supporting classes)

Added unit test for LocalAndSortedPriorRandomSearch

Removed the need to sample the prior to get the co-domain of the prior

Changed input kwargs in facade

Confirmed issue combining inactive with Normal / Beta hyperparameters

Added beauitified example usage

And of course, right branch this time

Features

Important Note

Major changes

Minor Changes

To build and view

Does not

Version issues

Issues

Description

Steps/Code to Reproduce

Versions

Required for RC

Discussions

Description

Description

Steps/Code to Reproduce

Expected Results

Actual Results

Versions

Description

Specific application

An initial idea

Releases(v2.0.0-beta.1)

v2.0.0-beta.1(Jan 7, 2023)

v2.0.0-alpha.2(Oct 26, 2022)

Bugfixes

Minor

v2.0.0-alpha.1(Oct 12, 2022)

Big Changes

New Features

Fixes

Optimization Changes

Code Related

v1.4.0(Jul 14, 2022)

Features

Improvements

Interface changes

v1.3.4(Jun 23, 2022)

v1.3.3(May 19, 2022)

v1.3.2(May 5, 2022)

v1.3.1(Apr 22, 2022)

v1.3.0(Apr 19, 2022)

Features

Minor Changes

Code-Quality

Documentation

v1.2.0(Feb 17, 2022)

Features

Major Changes

Examples

Documentation

Workflow Clean-up

Minor Changes

v1.1.1(Jan 27, 2022)

Minor Changes