Benchmarks for Model-Based Optimization

Last update: Dec 20, 2022

Related tags

Overview

Design-Bench

Design-Bench is a benchmarking framework for solving automatic design problems that involve choosing an input that maximizes a black-box function. This type of optimization is used across scientific and engineering disciplines in ways such as designing proteins and DNA sequences with particular functions, chemical formulas and molecule substructures, the morphology and controllers of robots, and many more applications.

These applications have significant potential to accelerate research in biochemistry, chemical engineering, materials science, robotics and many other disciplines. We hope this framework serves as a robust platform to drive these applications and create widespread excitement for model-based optimization.

Offline Model-Based Optimization

The goal of model-based optimization is to find an input x that maximizes an unknown black-box function f. This function is frequently difficulty or costly to evaluate---such as requiring wet-lab experiments in the case of protein design. In these cases, f is described by a set of function evaluations: D = {(x_0, y_0), (x_1, y_1), ... (x_n, y_n)}, and optimization is performed without querying f on new data points.

Installation

Design-Bench can be installed with the complete set of benchmarks via our pip package.

conda create -n mbo -c conda-forge rdkit
pip install design-bench[all]==2.0.20
pip install morphing-agents==1.5.1

Alternatively, if you do not have MuJoCo, you may opt for a minimal install.

conda create -n mbo -c conda-forge rdkit
pip install design-bench==2.0.20

Available Tasks

In the below table, we list the supported datasets and objective functions for model-based optimization, where a ✅ indicates that a particular combination has been tested and is available for download from our server.

Dataset \ Oracle	Exact	Gaussian Process	Random Forest	Fully Connected	LSTM	ResNet	Transformer
TF Bind 8	✅	✅	✅	✅	✅	✅	✅
GFP		✅	✅	✅	✅	✅	✅
ChEMBL		✅	✅	✅	✅	✅	✅
UTR		✅	✅	✅	✅	✅	✅
Hopper Controller	✅	✅	✅	✅
Superconductor		✅	✅	✅
Ant Morphology	✅	✅	✅	✅
D'Kitty Morphology	✅	✅	✅	✅

Combinations of datasets and oracles that are not available for download from our server are automatically trained on your machine on task creation. This currently only affects approximate oracles on user-defined MBO tasks. Below we provide the preferred oracle for each task, as well as meta data such as the number of data points measured.

Task Name	Dataset	Oracle	Dataset Size	Spearman's ρ
TFBind8-Exact-v0	TF Bind 8	Exact	65792
GFP-Transformer-v0	GFP	Transformer	56086	0.8497
ChEMBL-ResNet-v0	ChEMBL	ResNet	40516	0.3208
UTR-ResNet-v0	UTR	ResNet	280000	0.8617
HopperController-Exact-v0	Hopper Controller	Exact	3200
Superconductor-RandomForest-v0	Superconductor	Random Forest	21263	0.9155
AntMorphology-Exact-v0	Ant Morphology	Exact	25009
DKittyMorphology-Exact-v0	D'Kitty Morphology	Exact	25009

Performance Of Baselines

We benchmark a set of 9 methods for solving offline model-based optimization problems. Performance is reported in normalized form, where the 100th percentile score of 128 candidate designs is evaluated and normalized such that a 1.0 corresponds to performance equivalent to the best performing design in the full unobserved dataset assoctated with each model-based optimization task. A 0.0 corresponds to performance equivalent to the worst performing design in the full unobserved dataset. In circumstances where an exact oracle is not available, this full unobserved dataset is used for training the approximate oracle that is used for evaluation of candidate designs proposed by each method. The symbol ± indicates the empirical standard deviation of reported performance across 8 trials.

Method \ Task	GFP	TF Bind 8	UTR	ChEMBL
Auto. CbAS	0.865 ± 0.000	0.910 ± 0.044	0.650 ± 0.006	0.470 ± 0.000
CbAS	0.865 ± 0.000	0.927 ± 0.051	0.650 ± 0.002	0.517 ± 0.055
BO-qEI	0.254 ± 0.352	0.798 ± 0.083	0.659 ± 0.000	0.333 ± 0.035
CMA-ES	0.054 ± 0.002	0.953 ± 0.022	0.666 ± 0.004	0.350 ± 0.017
Grad.	0.864 ± 0.001	0.977 ± 0.025	0.639 ± 0.009	0.360 ± 0.029
Grad. Min	0.864 ± 0.000	0.984 ± 0.012	0.647 ± 0.007	0.361 ± 0.004
Grad. Mean	0.864 ± 0.000	0.986 ± 0.012	0.647 ± 0.005	0.373 ± 0.013
MINs	0.865 ± 0.001	0.905 ± 0.052	0.649 ± 0.004	0.473 ± 0.057
REINFORCE	0.865 ± 0.000	0.948 ± 0.028	0.646 ± 0.005	0.459 ± 0.036

Performance On Discrete Tasks.

Method \ Task	Superconductor	Ant Morphology	D'Kitty Morphology	Hopper Controller
Auto. CbAS	0.421 ± 0.045	0.884 ± 0.046	0.906 ± 0.006	0.137 ± 0.005
CbAS	0.503 ± 0.069	0.879 ± 0.032	0.892 ± 0.008	0.141 ± 0.012
BO-qEI	0.402 ± 0.034	0.820 ± 0.000	0.896 ± 0.000	0.550 ± 0.118
CMA-ES	0.465 ± 0.024	1.219 ± 0.738	0.724 ± 0.001	0.604 ± 0.215
Grad.	0.518 ± 0.024	0.291 ± 0.023	0.874 ± 0.022	1.035 ± 0.482
Grad. Min	0.506 ± 0.009	0.478 ± 0.064	0.889 ± 0.011	1.391 ± 0.589
Grad. Mean	0.499 ± 0.017	0.444 ± 0.081	0.892 ± 0.011	1.586 ± 0.454
MINs	0.469 ± 0.023	0.916 ± 0.036	0.945 ± 0.012	0.424 ± 0.166
REINFORCE	0.481 ± 0.013	0.263 ± 0.032	0.562 ± 0.196	-0.020 ± 0.067

Performance On Continuous Tasks.

Reproducing Baseline Performance

In order to reproduce this table, you must first install the implementation of the baseline algorithms.

git clone https://github.com/brandontrabucco/design-baselines
conda env create -f design-baselines/environment.yml
conda activate design-baselines

You may then run the following series of commands in a bash terminal using the command-line interface exposed in design-baselines. Also, please ensure that the conda environment design-baselines is activated in the bash session that you run these commands from in order to access the design-baselines command-line interface.

# set up machine parameters
NUM_CPUS=32
NUM_GPUS=8

for TASK_NAME in \
    gfp \
    tf-bind-8 \
    utr \
    chembl \
    superconductor \
    ant \
    dkitty \
    hopper; do
    
  for ALGORITHM_NAME in \
      autofocused-cbas \
      cbas \
      bo-qei \
      cma-es \
      gradient-ascent \
      gradient-ascent-min-ensemble \
      gradient-ascent-mean-ensemble \
      mins \
      reinforce; do
  
    # launch several model-based optimization algorithms using the command line interface
    # for example: 
    # (design-baselines) [email protected]:~/$ cbas gfp \
    #                                        --local-dir ~/db-results/cbas-gfp \
    #                                        --cpus 32 \
    #                                        --gpus 8 \
    #                                        --num-parallel 8 \
    #                                        --num-samples 8
    $ALGORITHM_NAME $TASK_NAME \
      --local-dir ~/db-results/$ALGORITHM_NAME-$TASK_NAME \
      --cpus $NUM_CPUS \
      --gpus $NUM_GPUS \
      --num-parallel 8 \
      --num-samples 8
    
  done
  
done

# generate the main performance table of the paper
design-baselines make-table --dir ~/db-results/ --percentile 100th

# generate the performance tables in the appendix
design-baselines make-table --dir ~/db-results/ --percentile 50th
design-baselines make-table --dir ~/db-results/ --percentile 100th --no-normalize

These commands will run several model-based optimization algorithms (such as CbAS) contained in design-baselines on all tasks released with the design-bench benchmark, and will then generate three performance tables from those results, and print a latex rendition of these performance tables to stdout.

The Train-Test Discrepency

For tasks where an exact numerical ground truth is not available for evaluating the performance of previously unseen candidate designs, we provide several families of approximate oracle models that have been trained using a larger held out dataset of designs x and corresponding scores y.

Using a learned oracle for evaluation and training an MBO method using real data creates a train-test discrepency. This discrepency can be avoided by relabelling the y values in an offline MBO dataset with the predictions of the learned oracle, which is controlled by the following parameter when building a task.

import design_bench

# instantiate the task using y values generated from the learned oracle
task = design_bench.make('GFP-Transformer-v0', relabel=True)

# instantiate the task using y values generated from real experiments
task = design_bench.make('GFP-Transformer-v0', relabel=False)

Task API

Design-Bench tasks share a common interface specified in design_bench/task.py, which exposes a set of input designs task.x and a set of output predictions task.y. In addition, the performance of a new set of input designs (such as those output from a model-based optimization algorithm) can be found using y = task.predict(x).

import design_bench
task = design_bench.make('TFBind8-Exact-v0')

def solve_optimization_problem(x0, y0):
    return x0  # solve a model-based optimization problem

# solve for the best input x_star and evaluate it
x_star = solve_optimization_problem(task.x, task.y)
y_star = task.predict(x_star)

Many datasets of interest to practitioners are too large to load in memory all at once, and so the task interface defines an several iterables that load samples from the dataset incrementally.

import design_bench
task = design_bench.make('TFBind8-Exact-v0')

for x, y in task:
   pass  # train a model here
   
for x, y in task.iterate_batches(32):
   pass  # train a model here
   
for x, y in task.iterate_samples():
   pass  # train a model here

Certain optimization algorithms require a particular input format, and so tasks support normalization of both task.x and task.y, as well as conversion of task.x from discrete tokens to the logits of a categorical probability distribution---needed when optimizing x with a gradient-based model-based optimization algorithm.

import design_bench
task = design_bench.make('TFBind8-Exact-v0')

# convert x to logits of a categorical probability distribution
task.map_to_logits()
discrete_x = task.to_integers(task.x)

# normalize the inputs to have zero mean and unit variance
task.map_normalize_x()
original_x = task.denormalize_x(task.x)

# normalize the outputs to have zero mean and unit variance
task.map_normalize_y()
original_y = task.denormalize_y(task.y)

# remove the normalization applied to the outputs
task.map_denormalize_y()
normalized_y = task.normalize_y(task.y)

# remove the normalization applied to the inputs
task.map_denormalize_x()
normalized_x = task.normalize_x(task.x)

# convert x back to integers
task.map_to_integers()
continuous_x = task.to_logits(task.x)

Each task provides access to the model-based optimization dataset used to learn the oracle (where applicable) as well as the oracle itself, which includes metadata for how it was trained (where applicable). These provide fine-grain control over the data distribution for model-based optimization.

import design_bench
task = design_bench.make('GFP-GP-v0')

# an instance of the DatasetBuilder class from design_bench.datasets.dataset_builder
dataset = task.dataset

# modify the distribution of the task dataset
dataset.subsample(max_samples=10000, 
                  distribution="uniform",
                  min_percentile=10, 
                  max_percentile=90)

# an instance of the OracleBuilder class from design_bench.oracles.oracle_builder
oracle = task.oracle

# check how the model was fit
print(oracle.params["rank_correlation"],
      oracle.params["model_kwargs"],
      oracle.params["split_kwargs"])

Dataset API

Datasets provide a model-based optimization algorithm with information about the black-box function, and are used in design bench to fit approximate oracle models when an exact oracle is not available. All datasets inherit from the DatasetBuilder class defined in design_bench.datasets.dataset_builder.

All datasets implement methods for modifying the format and distribution of the dataset, including normalization, subsampling, relabelling the outputs, and (for discrete datasets) converting discrete inputs to real-valued. There are also special methods for splitting the dataset into a training and validation set.

Display code snippet

from design_bench.datasets.discrete.gfp_dataset import GFPDataset
dataset = GFPDataset()

# convert x to logits of a categorical probability distribution
dataset.map_to_logits()
discrete_x = dataset.to_integers(dataset.x)

# normalize the inputs to have zero mean and unit variance
dataset.map_normalize_x()
original_x = dataset.denormalize_x(dataset.x)

# normalize the outputs to have zero mean and unit variance
dataset.map_normalize_y()
original_y = dataset.denormalize_y(dataset.y)

# remove the normalization applied to the outputs
dataset.map_denormalize_y()
normalized_y = dataset.normalize_y(dataset.y)

# remove the normalization applied to the inputs
dataset.map_denormalize_x()
normalized_x = dataset.normalize_x(dataset.x)

# convert x back to integers
dataset.map_to_integers()
continuous_x = dataset.to_logits(dataset.x)

# modify the distribution of the dataset
dataset.subsample(max_samples=10000, 
                   distribution="uniform",
                   min_percentile=10, 
                   max_percentile=90)

# change the outputs as a function of their old values
dataset.relabel(lambda x, y: y ** 2 - 2.0 * y)

# split the dataset into a validation set
training, validation = dataset.split(val_fraction=0.1)

If you would like to define your own dataset for use with design-bench, you can directly instantiate a continuous dataset or a discrete dataset depending on the input format you are using. The DiscreteDataset class and ContinuousDataset are built with this in mind, and accept both two numpy arrays containing inputs x outputs y.

Display code snippet

from design_bench.datasets.discrete_dataset import DiscreteDataset
from design_bench.datasets.continuous_dataset import ContinuousDataset
import numpy as np

# create dummy inputs and outputs for model-based optimization
x = np.random.randint(500, size=(5000, 43))
y = np.random.uniform(size=(5000, 1))

# create a discrete dataset for those inputs and outputs
dataset = DiscreteDataset(x, y)

# create dummy inputs and outputs for model-based optimization
x = np.random.uniform(size=(5000, 871))
y = np.random.uniform(size=(5000, 1))

# create a continuous dataset for those inputs and outputs
dataset = ContinuousDataset(x, y)

In the event that you are using a dataset that is saved to a set of sharded numpy files (ending in .npy), you may also create dataset by providing a list of shard files representing using the DiskResource class. The DiscreteDataset class and ContinuousDataset accept two lists of sharded inputs x and outputs y represented by DiskResource objects.

Display code snippet

from design_bench.disk_resource import DiskResource
from design_bench.datasets.discrete_dataset import DiscreteDataset
from design_bench.datasets.continuous_dataset import ContinuousDataset
import os
import numpy as np

# list the disk resource for each shard
os.makedirs("new_dataset/")
x = [DiskResource("new_dataset/shard-x-0.npy"), 
     DiskResource("new_dataset/shard-x-1.npy")]
y = [DiskResource("new_dataset/shard-y-0.npy"), 
     DiskResource("new_dataset/shard-y-1.npy")]

# create dummy inputs and outputs for model-based optimization
xs = np.random.randint(500, size=(5000, 43))
ys = np.random.uniform(size=(5000, 1))

# save the dataset to a set of shard files
np.save("new_dataset/shard-x-0.npy", xs[:3000])
np.save("new_dataset/shard-x-1.npy", xs[3000:])
np.save("new_dataset/shard-y-0.npy", ys[:3000])
np.save("new_dataset/shard-y-1.npy", ys[3000:])

# create a discrete dataset for those inputs and outputs
dataset = DiscreteDataset(x, y)

# create dummy inputs and outputs for model-based optimization
xs = np.random.uniform(size=(5000, 871))
ys = np.random.uniform(size=(5000, 1))

# save the dataset to a set of shard files
np.save("new_dataset/shard-x-0.npy", xs[:3000])
np.save("new_dataset/shard-x-1.npy", xs[3000:])
np.save("new_dataset/shard-y-0.npy", ys[:3000])
np.save("new_dataset/shard-y-1.npy", ys[3000:])

# create a continuous dataset for those inputs and outputs
dataset = ContinuousDataset(x, y)

Oracle API

Oracles provide a way of measuring the performance of candidate solutions to a model-based optimization problem, found by a model-based optimization algorithm, without having to perform additional real-world experiments. To this end, oracle implement a prediction function oracle.predict(x) that takes a set of designs and makes a prediction about their performance. The goal of model-based optimization is to maximize the predictions of the oracle.

Display code snippet

from design_bench.datasets.discrete.gfp_dataset import GFPDataset
from design_bench.oracles.tensorflow import TransformerOracle

# create a dataset and a noisy oracle
dataset = GFPDataset()
oracle = TransformerOracle(dataset, noise_std=0.1)

def solve_optimization_problem(x0, y0):
    return x0  # solve a model-based optimization problem

# evaluate the performance of the solution x_star
x_star = solve_optimization_problem(dataset.x, dataset.y)
y_star = oracle.predict(x_star)

In order to handle when an exact ground truth is unknown or not tractable to evaluate, Design-Bench provides a set of approximate oracles including a Gaussian Process, Random Forest, and several deep neural network architectures specialized to particular data modalities. These approximate oracles may have the following parameters.

Display code snippet

from design_bench.datasets.discrete.gfp_dataset import GFPDataset
from design_bench.oracles.tensorflow import TransformerOracle

# parameters for the transformer architecture
model_kwargs=dict(
    hidden_size=64,
    feed_forward_size=256,
    activation='relu',
    num_heads=2,
    num_blocks=4,
    epochs=20,
    shuffle_buffer=60000,
    learning_rate=0.0001,
    dropout_rate=0.1)

# parameters for building the validation set
split_kwargs=dict(
    val_fraction=0.1,
    subset=None,
    shard_size=5000,
    to_disk=True,
    disk_target="gfp/split",
    is_absolute=False)
    
# create a transformer oracle for the GFP dataset
dataset = GFPDataset()
oracle = TransformerOracle(
    dataset, 
    noise_std=0.0,
    
    # parameters for ApproximateOracle subclasses
    disk_target="new_model.zip",
    is_absolute=True,
    fit=True,
    max_samples=None,
    distribution=None,
    max_percentile=100,
    min_percentile=0,
    model_kwargs=model_kwargs,
    split_kwargs=split_kwargs)

def solve_optimization_problem(x0, y0):
    return x0  # solve a model-based optimization problem

# evaluate the performance of the solution x_star
x_star = solve_optimization_problem(dataset.x, dataset.y)
y_star = oracle.predict(x_star)

Defining New MBO Tasks

New model-based optimization tasks are simple to create and register with design-bench. By subclassing either DiscreteDataset or ContinuousDataset, and providing either a pair of numpy arrays containing inputs and outputs, or a pair of lists of DiskResource shards containing inputs and outputs, you can define your own model-based optimization dataset class. Once a custom dataset class is created, you can register it as a model-based optimization task by choosing an appropriate oracle type, and making a call to the register function. After doing so, subsequent calls to design_bench.make can find your newly registered model-based optimization task.

Display code snippet

from design_bench.datasets.continuous_dataset import ContinuousDataset
import design_bench
import numpy as np

# define a custom dataset subclass of ContinuousDataset
class QuadraticDataset(ContinuousDataset):

    def __init__(self, **kwargs):
    
        # define a set of inputs and outputs of a quadratic function
        x = np.random.normal(0.0, 1.0, (5000, 7))
        y = (x ** 2).sum(keepdims=True)
        
        # pass inputs and outputs to the base class
        super(QuadraticDataset, self).__init__(x, y, **kwargs)

# parameters used for building the validation set
split_kwargs=dict(
    val_fraction=0.1,
    subset=None,
    shard_size=5000,
    to_disk=True,
    disk_target="quadratic/split",
    is_absolute=True)

# parameters used for building the model
model_kwargs=dict(
    hidden_size=512,
    activation='relu',
    num_layers=2,
    epochs=5,
    shuffle_buffer=5000,
    learning_rate=0.001)

# keyword arguments for building the dataset
dataset_kwargs=dict(
    max_samples=None,
    distribution=None,
    max_percentile=80,
    min_percentile=0)

# keyword arguments for training FullyConnected oracle
oracle_kwargs=dict(
    noise_std=0.0,
    max_samples=None,
    distribution=None,
    max_percentile=100,
    min_percentile=0,
    split_kwargs=split_kwargs,
    model_kwargs=model_kwargs)

# register the new dataset with design_bench
design_bench.register(
    'Quadratic-FullyConnected-v0', QuadraticDataset,
    'design_bench.oracles.tensorflow:FullyConnectedOracle',
    dataset_kwargs=dataset_kwargs, oracle_kwargs=oracle_kwargs)
                 
# build the new task (and train a model)         
task = design_bench.make("Quadratic-FullyConnected-v0")

def solve_optimization_problem(x0, y0):
    return x0  # solve a model-based optimization problem

# evaluate the performance of the solution x_star
x_star = solve_optimization_problem(task.x, task.y)
y_star = task.predict(x_star)

Citation

Thanks for using our benchmark, and please cite our paper!

@misc{
    trabucco2021designbench,
    title={Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization},
    author={Brandon Trabucco and Aviral Kumar and Xinyang Geng and Sergey Levine},
    year={2021},
    url={https://openreview.net/forum?id=cQzf26aA3vM}
}

Comments

Difficulties in installing with pip install design-bench[all]
Hi,

I really enjoyed reading your paper in https://openreview.net/forum?id=cQzf26aA3vM and I am trying to use your code.

However, I am having difficulties when I try to install the full version of your repository. Could you provide guidance for resolving this issue?

I tried running "pip install design-bench[all]" as the README.md suggested. However, this fails after a long time of install with many warnings of:

pip is looking at multiple versions of XXX package to determine which version is compatible with other requirements.

The final error message looks like:

ImportError: cannot import name 'chardet2' from 'requests.packages' (/tmp/pip-install-r9ss5d00/requests_8bb0b6f4c8244f9ca00e3f1fe1ca7bd9/requests/packages/init.py)

I also tried downloading the files from https://sites.google.com/view/design-bench/ and running "python setup.py install easy_install design-bench[all]". For this case, I run into the following error:

error: mujoco-py 2.0.2.13 is installed but mujoco_py<2.0,>=1.50 is required by {'gym'}

Running "pip install design-bench" works fine!

Thank you very much!
opened by sungsoo-ahn 4
oracle.expect_normalized_y for TFBind8

Hi,

I am not sure as to why for various oracles in TFBind8, the field task.oracle.expect_normalized_y is set to True. I assumed that this means that we should expect that the oracle's predictions will lie in the same range as the normalised y variable, but this does not appear to be the case.

If I histogram task.oracle.internal_dataset.y then I get the following:

(I assume internal_dataset is exactly the dataset used to train the approximate oracle, without any extra preprocessing.)

It can also be seen here, if we plot y_i (normalised) against oracle.predict(x_i) for all values in the training set. It looks like the range of predictions for the oracle lie in [0,1]:

Can you clarify whether this is a bug or whether I have misunderstood what this field is meant to imply? Thanks!

opened by christopher-beckham 1
Bump tensorflow from 2.4.1 to 2.5.0
Bumps tensorflow from 2.4.1 to 2.5.0.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.5.0

Release 2.5.0

Major Features and Improvements

Support for Python3.9 has been added.

tf.data:

tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.

tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).

tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.

Options returned by tf.data.Dataset.options() are no longer mutable.

tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().

tf.lite

Enabled the new MLIR-based quantization backend by default

The new backend is used for 8 bits full integer post-training quantization

The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)

Set experimental_new_quantizer in tf.lite.TFLiteConverter to False to disable this change

tf.keras

tf.keras.metrics.AUC now support logit predictions.

Enabled a new supported input type in Model.fit, tf.keras.utils.experimental.DatasetCreator, which takes a callable, dataset_fn. DatasetCreator is intended to work across all tf.distribute strategies, and is the only input type supported for Parameter Server strategy.

tf.distribute

tf.distribute.experimental.ParameterServerStrategy now supports training with Keras Model.fit when used with DatasetCreator.

Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.

TPU embedding support

Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.

PluggableDevice

Third-party devices can now connect to TensorFlow as plug-ins through StreamExecutor C API. and PluggableDevice interface.

Add custom ops and kernels through kernel and op registration C API.

Register custom graph optimization passes with graph optimization C API.

oneAPI Deep Neural Network Library (oneDNN) CPU performance optimizations from Intel-optimized TensorFlow are now available in the official x86-64 Linux and Windows builds.

They are off by default. Enable them by setting the environment variable TF_ENABLE_ONEDNN_OPTS=1.

We do not recommend using them in GPU systems, as they have not been sufficiently tested with GPUs yet.

TensorFlow pip packages are now built with CUDA11.2 and cuDNN 8.1.0

Breaking Changes

The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

Bug Fixes and Other Changes

tf.keras:

Preprocessing layers API consistency changes:

StringLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization.

IntegerLookup added output_mode, sparse, and pad_to_max_tokens arguments with same semantics as TextVectorization. Renamed max_values, oov_value and mask_value to max_tokens, oov_token and mask_token to align with StringLookup and TextVectorization.

TextVectorization default for pad_to_max_tokens switched to False.

CategoryEncoding no longer supports adapt, IntegerLookup now supports equivalent functionality. max_tokens argument renamed to num_tokens.

Discretization added num_bins argument for learning bins boundaries through calling adapt on a dataset. Renamed bins argument to bin_boundaries for specifying bins without adapt.

Improvements to model saving/loading:

model.load_weights now accepts paths to saved models.

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.5.0

Breaking Changes

The TF_CPP_MIN_VLOG_LEVEL environment variable has been renamed to to TF_CPP_MAX_VLOG_LEVEL which correctly describes its effect.

Known Caveats

Major Features and Improvements

TPU embedding support

Added profile_data_directory to EmbeddingConfigSpec in _tpu_estimator_embedding.py. This allows embedding lookup statistics gathered at runtime to be used in embedding layer partitioning decisions.

tf.keras.metrics.AUC now support logit predictions.

Creating tf.random.Generator under tf.distribute.Strategy scopes is now allowed (except for tf.distribute.experimental.CentralStorageStrategy and tf.distribute.experimental.ParameterServerStrategy). Different replicas will get different random-number streams.

tf.data:

tf.data service now supports strict round-robin reads, which is useful for synchronous training workloads where example sizes vary. With strict round robin reads, users can guarantee that consumers get similar-sized examples in the same step.

tf.data service now supports optional compression. Previously data would always be compressed, but now you can disable compression by passing compression=None to tf.data.experimental.service.distribute(...).

tf.data.Dataset.batch() now supports num_parallel_calls and deterministic arguments. num_parallel_calls is used to indicate that multiple input batches should be computed in parallel. With num_parallel_calls set, deterministic is used to indicate that outputs can be obtained in the non-deterministic order.

Options returned by tf.data.Dataset.options() are no longer mutable.

tf.data input pipelines can now be executed in debug mode, which disables any asynchrony, parallelism, or non-determinism and forces Python execution (as opposed to trace-compiled graph execution) of user-defined functions passed into transformations such as map. The debug mode can be enabled through tf.data.experimental.enable_debug_mode().

tf.lite

Enabled the new MLIR-based quantization backend by default

The new backend is used for 8 bits full integer post-training quantization

The new backend removes the redundant rescales and fixes some bugs (shared weight/bias, extremely small scales, etc)

... (truncated)

Commits

a4dfb8d Merge pull request #49124 from tensorflow/mm-cherrypick-tf-data-segfault-fix-...

2107b1d Merge pull request #49116 from tensorflow-jenkins/version-numbers-2.5.0-17609

16b8139 Update snapshot_dataset_op.cc

86a0d86 Merge pull request #49126 from geetachavan1/cherrypicks_X9ZNY

9436ae6 Merge pull request #49128 from geetachavan1/cherrypicks_D73J5

6b2bf99 Validate that a and b are proper sparse tensors

c03ad1a Ensure validation sticks in banded_triangular_solve_op

12a6ead Merge pull request #49120 from geetachavan1/cherrypicks_KJ5M9

b67f5b8 Merge pull request #49118 from geetachavan1/cherrypicks_BIDTR

a13c0ad [tf.data][cherrypick] Fix snapshot segfault when using repeat and prefecth

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

normalize and denommalize issue

Hi Brandon,

I recently install and use the dataset from your package. Thanks for your work to build this benchmark.

I have a problem with the function task.normalize_x.

# load task
task = design_bench.make('AntMorphology-Exact-v0', relabel=False)
# get the top 128 features
aa = task.x[np.argsort(np.squeeze(task.y))[-128:]]
# predict the labels
r1 = np.squeeze(task.predict(aa))

# normalize and denormalize features
bb = task.normalize_x(aa)
cc = task.denormalize_x(bb)
# predict the labels again
r2 = np.squeeze(task.predict(cc))

print(np.max(r1), np.max(r2)) #--> 198.7532   406.76566
print(np.where((aa-cc)>0.001))  #--> (array([], dtype=int64), array([], dtype=int64))

The output shows that normalization and denormalization don't change the features but predictions are quite different. Is there anything wrong with my codes? I feel it's a trivial issue. But I cannot figure out where the problem is.

Nathan

opened by Nathan-zh 4

Assessing uncertainty quantification quality metrics using the `design-bench` benchmarks

I'm considering using this for some simple tests of a few different uncertainty quantification quality metrics to see which ones are better predictors for how successful an adaptive design scheme will be.

From a very black box standpoint, what this requires is y_true, y_pred, and sigma (true, predicted, and uncertainty, resp.) and a "notion of best" for the adaptive design task.

Does that seem like something feasible/easy to implement with this repository? Or do you think it would be better to look elsewhere or start from scratch?

opened by sgbaird 2
Summary of the characteristics of the datasets?

For example, dimensionality, number of numerical vs. categorical features, and if possible, some notion of complexity for each of the tasks (an example would be a comparison against random search performance)

opened by sgbaird 2

AttributeError: module 'numpy' has no attribute 'loads'

Hi,

After following the instructions in the readme (creating a new environment mbo with the required dependencies) and trying to create a new task, I get the following error:

task = design_bench.make('Superconductor-GP-v0')
/mnt/home/envs/mbo/lib/python3.9/site-packages/sklearn/base.py:329: UserWarning: Trying to unpickle estimator GaussianProcessRegressor from version 0.23.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/home/github/design-bench/design_bench/registration.py", line 327, in make
    return registry.make(task_name, dataset_kwargs=dataset_kwargs,
  File "/mnt/home/github/design-bench/design_bench/registration.py", line 155, in make
    return self.spec(task_name).make(
  File "/mnt/home/github/design-bench/design_bench/registration.py", line 109, in make
    return Task(self.dataset, self.oracle,
  File "/mnt/home/github/design-bench/design_bench/task.py", line 263, in __init__
    oracle = import_name(oracle)(dataset, **kwargs)
  File "/mnt/home/github/design-bench/design_bench/oracles/sklearn/gaussian_process_oracle.py", line 91, in __init__
    super(GaussianProcessOracle, self).__init__(
  File "/mnt/home/github/design-bench/design_bench/oracles/approximate_oracle.py", line 300, in __init__
    self.params = self.load_params(self.resource.disk_target)
  File "/mnt/home/github/design-bench/design_bench/oracles/approximate_oracle.py", line 394, in load_params
    rank_correlation = np.loads(file.read())
  File "/mnt/home/envs/mbo/lib/python3.9/site-packages/numpy/__init__.py", line 315, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'loads'

It seems like loads does not exist in this version of Numpy ('1.22.3'). Changing the method call to np.load(...) also does not fix the issue and throws another error.

Thanks.

opened by christopher-beckham 2

Releases(v2_0_20)

v2_0_20(Oct 22, 2021)

This is a patch that updates the downloader for the SMILES vocab file, and also updates the requirements for install.
Source code(tar.gz)
Source code(zip)
v2_0_18(Sep 5, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_17(Sep 5, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_16(Sep 1, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_15(Aug 31, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_14(Aug 26, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_13(Aug 26, 2021)

Release For NeurIPS Round 2:

Adding two new MBO tasks with exact ground truths. Specifically, TFBind10, which is an optimization problem over length 10 DNA nucleotide sequences, and CIFAR-NAS, which is an optimization problem over the architecture of CNN models on CIFAR-10.
Source code(tar.gz)
Source code(zip)
v2_0_12(Jul 8, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_11(Jun 7, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_10(Jun 7, 2021)

Source code(tar.gz)
Source code(zip)
v2_0_8(Jun 5, 2021)

Source code(tar.gz)
Source code(zip)
v1_6(Mar 26, 2021)

Fix for the oracle score function for TFBind8
Source code(tar.gz)
Source code(zip)
v1_5(Mar 25, 2021)

Adding two new discrete MBO tasks, inspired by work from https://arxiv.org/pdf/2006.03227v2.pdf.
Source code(tar.gz)
Source code(zip)
v1_4(Jan 25, 2021)

Adding new parameters to every task.
Source code(tar.gz)
Source code(zip)
v1_3(Jan 25, 2021)

Update default settings, and change the stochasticity of MuJoCo environments.
Source code(tar.gz)
Source code(zip)
v1_2(Nov 17, 2020)

Additional GFP Oracle
Source code(tar.gz)
Source code(zip)
v1_1(Oct 14, 2020)

Source code(tar.gz)
Source code(zip)
v1(Oct 4, 2020)

Source code(tar.gz)
Source code(zip)

Owner

Brandon Trabucco

Musician & Machine Learning PhD Student @ CMU, previously an Undergraduate @ BerkeleyAI

GitHub Repository

Benchmarks for Model-Based Optimization

Related tags

Overview

Design-Bench

Offline Model-Based Optimization

Installation

Available Tasks

Performance Of Baselines

Reproducing Baseline Performance

The Train-Test Discrepency

Task API

Dataset API

Oracle API

Defining New MBO Tasks

Citation

Comments

Difficulties in installing with pip install design-bench[all]

oracle.expect_normalized_y for TFBind8

Bump tensorflow from 2.4.1 to 2.5.0

TensorFlow 2.5.0

Release 2.5.0

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

Release 2.5.0

Breaking Changes

Known Caveats

Major Features and Improvements

normalize and denommalize issue

Assessing uncertainty quantification quality metrics using the `design-bench` benchmarks

Summary of the characteristics of the datasets?

AttributeError: module 'numpy' has no attribute 'loads'

Releases(v2_0_20)

v2_0_20(Oct 22, 2021)

v2_0_18(Sep 5, 2021)

v2_0_17(Sep 5, 2021)

v2_0_16(Sep 1, 2021)

v2_0_15(Aug 31, 2021)

v2_0_14(Aug 26, 2021)

v2_0_13(Aug 26, 2021)

v2_0_12(Jul 8, 2021)

v2_0_11(Jun 7, 2021)

v2_0_10(Jun 7, 2021)

v2_0_8(Jun 5, 2021)

v1_6(Mar 26, 2021)

v1_5(Mar 25, 2021)

v1_4(Jan 25, 2021)

v1_3(Jan 25, 2021)

v1_2(Nov 17, 2020)

v1_1(Oct 14, 2020)

v1(Oct 4, 2020)

Owner

Brandon Trabucco

Ego4d dataset repository. Download the dataset, visualize, extract features & example usage of the dataset

GAN-generated image detection based on CNNs

PyTorch implementation of VAGAN: Visual Feature Attribution Using Wasserstein GANs

HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

Aws-machine-learning-university-accelerated-tab - Machine Learning University: Accelerated Tabular Data Class

AI-generated-characters for Learning and Wellbeing

Uni-Fold: Training your own deep protein-folding models

A Python framework for conversational search

Corruption Invariant Learning for Re-identification

Binary Stochastic Neurons in PyTorch

Code for "Optimizing risk-based breast cancer screening policies with reinforcement learning"

Gems & Holiday Package Prediction

PyTorch Implementation of PortaSpeech: Portable and High-Quality Generative Text-to-Speech

git《Tangent Space Backpropogation for 3D Transformation Groups》(CVPR 2021) GitHub:1]

A repository for benchmarking neural vocoders by their quality and speed.

Official codebase for ICLR oral paper Unsupervised Vision-Language Grammar Induction with Shared Structure Modeling

Python code for the paper How to scale hyperparameters for quickshift image segmentation

The Balloon Learning Environment - flying stratospheric balloons with deep reinforcement learning.

Learning Modified Indicator Functions for Surface Reconstruction

Official Implementation of CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback