This PR aims to add a Dockerfile for the poutyne.

@freud14 while this image gets successfully built, I'd like to further tone down the image size. At the moment it's about 6.35 GB which makes no sense as the base image itself is 2.79 GB. After using dive to inspect the image it turns out the COPY statement from the base image takes up around ~6.2GB.

I've never encountered this problem before, would be grateful if somebody else pitches in.

To build the 🐳 image use DOCKER_BUILDKIT=1 docker build -t poutyne .

This PR is going to introduce a fastai-like "train phase interface" and create the foundation for more complex and fine grained control over the training process.

The PR addresses and closes #15.

This PR adds the following

• Parameter spaces: LinSpace and CosineSpace lazily generate values in the specified domain.
• The TrainPhase class uses the parameter spaces and specifies how to configure the optimizer in each step. It return a dict of optimizer parameters.
• The TrainPolicy combines different TrainPhases and updates the optimizer according to the parameters returned by the TrainPolicys.

TODO

• [x] first implementation
• [x] tweak implementation
• [x] Store the state of the TrainPolicy and make it restartable
• [x] add "one cycle" policy
• [x] add "sgdr" policy
• [x] fully test everything (and switch from pytest back to UnitTest)
  • [x] spaces
  • [x] TrainPhase
  • [x] TrainPolicy
  • [x] integration test
• [x] docs
  • [x] write docstrings
  • [x] add to sphinx
  • [x] clean up sphinx warnings
• [x] add references to the papers
• [x] create a example notebook
• [x] clean up commit history

Possible extensions

• Different optimizers could be set for each phase
• More policies
• Set different learning rates for different part of the network

The logo should be inspired from pytorch's Logo and have a log (like a wood log y'know) in fire. The description of the project in the README should start with something like "Pytoune is a Keras-like interface for the PyTorch framework...." and should have SEO (search engine optimization) in mind. I don't know what it is possible for that in github/markdown. We should also includes lots of examples of what's possible with our framework.

Is there anyway to implement the functionality as described in this post. https://stackoverflow.com/questions/46464549/keras-custom-loss-function-accessing-current-input-pattern

Gist: implementing something like code below

    def custom_loss_wrapper(input_tensor):
        def custom_loss(y_true, y_pred):
            return K.binary_crossentropy(y_true, y_pred) + K.mean(input_tensor)
   return custom_loss

This can be called with loss=custom_loss_wrapper(model.input)

due to lack of predict examples, I am unable to use the model.predict correctly, there also exists a bug as: RuntimeError: Expected 4-dimensional input for 4-dimensional weight [256, 1, 1, 3], but got 3-dimensional input of size [31, 2, 132] instead I used :output=model.predict_on_batch(Singledata) and my Singledata is a array of size (31,2,128), float type

Hi,

I believe current tensorboard callback does not use tensorboard summaries the way they were intended.

Currently different metrics groups are stored as different runs (left-hand side of tensorboard).

Those are meant to represent different experiment run that you can compare against themselves. With current implementation this kind of bookkeeping is very difficult.

This is current implementation:

With current approach is hard to compare different experiment and number of 'run' entries on left is overwhelming.

This PR makes TensorBoard callback behave as its counterpart in keras. All metrics are stored within one run only. That way you can easily compare different experiments.

See second screenshot for comparison:

Now I can easily see how many experiments I have and I can easily compare them against each other. If I want to select some plots subset on the right I can selectively hide/show any chart and/or use regex query. No need to clutter the 'runs' list!

Dear sir: I came across an error when I installed poutyne with "pip install -U git+https://github.com/GRAAL-Research/[email protected]". I think the problem is in my torch.As the Error information said: "ERROR: Failed building wheel for torch". But I have no idea in solving this.So I hope you can provide some solution to me. The overall error feedback is in below. Thanks!

ERROR: Command errored out with exit status 1: command: 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"'; file='"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-wheel-f_6axmet' cwd: C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch
Complete output (30 lines): running bdist_wheel running build running build_deps Traceback (most recent call last): File "", line 1, in File "C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py", line 225, in setup(name="torch", version="0.1.2.post2", File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\site-packages\setuptools_init_.py", line 144, in setup return distutils.core.setup(**attrs) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 966, in run_commands self.run_command(cmd) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\site-packages\wheel\bdist_wheel.py", line 223, in run self.run_command('build') File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py", line 51, in run from tools.nnwrap import generate_wrappers as generate_nn_wrappers ModuleNotFoundError: No module named 'tools.nnwrap'

ERROR: Failed building wheel for torch Running setup.py clean for torch ERROR: Command errored out with exit status 1: command: 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"'; file='"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' clean --all cwd: C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch Complete output (2 lines): running clean error: [Errno 2] No such file or directory: '.gitignore'

ERROR: Failed cleaning build dir for torch Successfully built Poutyne Failed to build torch Installing collected packages: torch, Poutyne Running setup.py install for torch ... error ERROR: Command errored out with exit status 1: command: 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"'; file='"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\彭 张智computational AI\AppData\Local\Temp\pip-record-e8il4qu8\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\Include\torch' cwd: C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch
Complete output (23 lines): running install running build_deps Traceback (most recent call last): File "", line 1, in File "C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py", line 225, in setup(name="torch", version="0.1.2.post2", File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\site-packages\setuptools_init_.py", line 144, in setup return distutils.core.setup(**attrs) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\core.py", line 148, in setup dist.run_commands() File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 966, in run_commands self.run_command(cmd) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py", line 99, in run self.run_command('build_deps') File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py", line 51, in run from tools.nnwrap import generate_wrappers as generate_nn_wrappers ModuleNotFoundError: No module named 'tools.nnwrap' ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"'; file='"'"'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-install-fgpcp5ez\torch\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\彭张智computational AI\AppData\Local\Temp\pip-record-e8il4qu8\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\彭 张智computational ai\appdata\local\programs\python\python38-32\Include\torch' Check the logs for full command output. WARNING: You are using pip version 20.0.2; however, version 20.2.3 is available. You should consider upgrading via the 'c:\users\彭张智computational ai\appdata\local\programs\python\python38-32\python.exe -m pip install --upgrade pip' command.

The core modification was to create a new TensorData that would recursively fetch the proper index value and a function that can aggregate batches of output as one big output.

I disabled one test test_disable_batch_size_warning as I am not sure if it's required anymore. I modified _get_batch_size to account for this new input/output structure, but I am not sure what the original intention was.

TL;DR: some LRSchedulers must be able to update the learning rate (and other parameters) during the batches and not just at the end of an epoch. Should we implement this? What is the best way?

Long version:

I would love to see PyToune offer advanced learning rate schedules like 1 cycle policy!

PyTorch's _LRScheduler have a step() method which set the learning rate. The user is then responsible to call the step() method somewhere in the training loop. This normally happens at the end of each epoch, but it could also happen after each batch.

PyToune wraps PyTorch's _LRSchedulers via the Callback interface and calls the step() method on_epoch_end. This is great for convenience, but some LRSchedulers have to update the lr after each batch.

Here are a few options I see:

• We could update the PyTorchLRSchedulerWrapper and make it configurable when to trigger the step() call, and then use it something like this:

  CosineAnnealingLR(step_on_batch_end=True, ...)
  

• We could introduce batch versions of all LRSchedulers.

• I use a simple callback LRSetter that gets a list of learning rates (and momentum values) and sets the lr (and momentum) on each batch end.

Hi,

I added a gist with a couple of functions to convert a numpy array (or python object such as list) into a pytorch tensor. I couldn't find anything like this in the library except a function going the other way around poutyne.torch_to_numpy.

These functions are copied from the FastAI library (core module) and modified a bit. If this is useful, I could submit a PR.

Would it be wise to support multiple output? At least a pass through? I have a model that returns two outputs (prediction and attention weights). I had to do a small wrapper around the loss function as such

    def loss_function(y_pred, y_true):
        output, word_attention = y_pred
        return criterion(output, y_true)

    def acc(y_pred, y_true, *args):
        y_pred, _ = y_pred
        return metrics.bin_acc(y_pred, y_true, *args)

Which is totally normal considering that this is an uncommon output and there is no way to know how to calculate the loss or the metric in a generic way. Training was working just fine with this. But when it came to predict() it couldn't get through because of

https://github.com/GRAAL-Research/poutyne/blob/master/poutyne/framework/model.py#L406

So I experimented a little bit and came up with this function in utils.py

def _concat(obj):
    if isinstance(obj[0], tuple):
        return tuple([_concat(ele) for ele in zip(*obj)])
    else:
        return np.concatenate(obj)

Then in predict() I call it instead of np.concatenate directly

        pred_y = self.predict_generator(generator)
        return utils._concat(pred_y)

This will work as long as the model returns either a tuple or multiple values at once return (output1, output2) or return output1, output2.

I don't know if it's a reasonable expectation. All the tests are passing. Since it requires custom loss functions (and metrics), there doesn't seem to be any new tests to add.

Just wanted your thoughts on this.

Thanks

When training for a classification task, FScore is more relevant than accuracy. After training, when I select the best model, I often get a subpar model, since a bette model with higher FScore may be available, and no checkpoint were log for it

Either use FScore as default monitor metric for classification or additionnal task such as "unbalanced_classification".

At the moment I have to manually add the FScore Macro and max as monitor mode. Very easy to fix - I just don't see why someone with a normal classification task would select accuracy over FScore as target metric.

It would be nice to add more examples of usage. For instance, how to use the available callbacks, etc. The examples could be in both the doc and in an "examples" folder.

It could be nice to have some kind of way to add regularization like it is possible in Keras with the add_loss function. Maybe adding some optional list of losses in the module that would be summed up by the Module class?

This should includes wrappers for pytorch modules, which do not support initializers in their constructors.

Furthermore, there should be some kind of way to take a given pytorch module and initialize it with default and user-chosen intializers depending on the type of the modules inside the given module. For instance, say I have some module A which contains other modules and some of them are of type Linear. The functionality should take the module A as a parameters and initialize the Linear modules in some default or non-default way.