re: openjournals/joss-reviews#4953

There are a handful of leftover TODO notes that appear in the rendered documentation. Some of these might be reasonable to leave in for now if they refer to not-yet-implemented features or examples (e.g., the 4th & 5th ones below?), but I think the first 3 are probably worth resolving in order to make the documentation complete.

• [ ] "TODO: add cites" under the "Group 1" subheading in the "Categories of Algorithms" section of the Algorithm Evaluation and Parameter Optimization page
• [ ] "TODO: Link" in the Warning in the docstring for tpcp.clone
• [ ] "learn more TODO" in the docstring for tpcp.PureParameter/tpcp.PurePara
  • (also, "parameter" and "pure_paramter" appear here in snake case—should these be camel case (i.e., "Parameter" and "PureParamter" to refer to the annotation classes?)
• [ ] "Todo: Full dedicated example for PureParameter" under the "The Pipeline" heading in the GridSearchCV example
• [ ] "TODO: Make GridSearchCV aware of it!" near the end of the "Optimization Info" example

re: openjournals/joss-reviews#4953

Based on the Python version requirement in pyproject.toml https://github.com/mad-lab-fau/tpcp/blob/d88c875bd33f8c0b0ea9eb6bc8b55cd8802311ed/pyproject.toml#L16 and the PyPI classifiers that Poetry automatically generates for the packages listing based on this, tpcp appears to support Python 3.8, 3.9, 3.10, and 3.11. However, the CI tests are currently run only on Python 3.8, 3.9, and 3.10: https://github.com/mad-lab-fau/tpcp/blob/d88c875bd33f8c0b0ea9eb6bc8b55cd8802311ed/.github/workflows/test-and-lint.yml#L14-L15

I'm guessing may be because Pytorch doesn't yet support Python 3.11 (pytorch/pytorch#86566) and is used in some of the tests. However, since the core of tpcp itself seems to support Python 3.11 just fine (I've now run most of the examples with Python 3.11.0 without issue, though of course pip install 'tpcp[torch]' fails), Pytorch is an optional dependency, and the other optional dependencies (optuna & attrs) do support Python 3.11, it would be great to have CI tests run against that version if it's going to be supported.

You could do this by:

1. adding "3.11" to your python-version array in line 15 of your workflow file
2. editing your "Install dependencies" step from: https://github.com/mad-lab-fau/tpcp/blob/d88c875bd33f8c0b0ea9eb6bc8b55cd8802311ed/.github/workflows/test-and-lint.yml#L23-L28 to:

    - name: Install dependencies
      env:
        PYTHON_VERSION: ${{ matrix.python-version }}
      run: |
        python -m pip install --upgrade pip
        pip install poetry
        poetry config virtualenvs.in-project true
        poetry install -E optuna
        [[ "$PYTHON_VERSION" == "3.11" ]] || poetry install -E torch
   

3. editing line 3 of test_hash.py from: https://github.com/mad-lab-fau/tpcp/blob/84a3048ff54b5d6912068009d15c65fdc5eb4592/tests/test_hash.py#L3 to:

   torch = pytest.importorskip("torch")
   

Once Pytorch adds support for Python 3.11, you could simply revert steps 2 and 3.

I also think it would be worth adding a quick note to the Dev Setup section of the README saying that development requires python>=3.8,<3.11 since the instructions there include installing Pytorch.

If the CI tests end up revealing other, more extensive issues with Python 3.11 compatibility that I didn't run into, you could of course instead just reduce the maximum supported version in your pyproject.toml for the JOSS submission.

re: openjournals/joss-reviews#4953

Hey @AKuederle, thanks for creating such a useful package and documenting everything so extensively! I found the "User Guides" and "Examples" sections particularly helpful. I came across just a few minor issues with the docs. The following hyperlinked text leads to readthedocs 404 pages:

• On the General Concepts page:
  • "mutable handling" (second paragraph)
  • "optimization guide" (end of the "Parameters" section)
  • "our example" (end of the "Composite Parameters" section)
• On the Optimization and Training page:
  • "evaluation guide" (#3 under "Parameter Annotations")
  • "example" (before the Note in the "Parameter Annotations" section)

This allows to use set_params and get_params calls for objects that contain list of other objects as parameters.

Note, that we only support one "type" of list namly one that consists of a sequence of (name, nested_obj) tupels. This is similar to how sklearn Pipelines work, with the difference that we allow multiple different parameters to be independent composite keys.

In contrast to sklearn, tpcp is focusod on running not training algorithms. This means, we allow to return multiple results from a run call (by placing them as attributes on the instance). However, it is not easily possible to do the same with a call to self_optimize. While we only expect the optimized version of a algorithm to be the outcome of a self_optimize call, it might be helpfull to have access to further information about the training.

At the moment this is conflicting with our fundamental API design. Further, pipelines during optimization are often cloned multiple times. This means information stored outside the parameters usually does not find its way back to user space. So whatever solution we come up with should allow to conserve this information when optimizations are run within GridSearchCV (or similar).

Other libraries often solve this using callbacks that can be called within a the method to generate a log. I don't like that, as it is unclear, how we would pass such callbacks to the optimize method and further how we would get the values from the Callback Recorder (in particular if everything is called in a nested loop like a GridSearch).

Another option would be to specify one attribute with a fixed name that is allowed to be modified by the optimize method. As it is a fixed named attribute, higher level Wrapper could be made aware of it and collect it before the pipeline is cloned after an optimization.

A first version of that is implemented in #45

One remaining issue in the fundamental design of tpcp is that it was basically impossible to extract informations from the trainings process. This PR tests an approach, where it is possible to fill an "info" dictionary during a self_optimize (or run call). The Optimize method has been made aware of this object and stores it as a result. This way it can be accessed when running GridSearchCV or other methods.

At the moment pipelines can not be dataclasses due to the way we perform the initial checks (i.e. we run the checks in __init_subclass, which is called before the dataclass wrapper is even called.

Maybe we can find a solution for that, which is not horribly hacky.

#27 implements a generic optimizer base class for optuna based optimizers. However, it alwayes requires the user to create subclasses to create custom optimizers.

In the future it would be nice to provide a specific GrideSearch and GridSearchCV equivalent implementation. These would either just call the scorer or call cross_validate in their objective function.

A first set of examples that are the foundations for all further examples.

@richrobe I decided to go with ECG instead of sleep for now (I could just copy and paste stuff from the biosig exercise :) ) Might still make sense to add sleep stuff later when talking about ML/other more complicated algorithms. Other advantage of ECG was, that the datasets are small enough to directly commit to the repo.

closes #2

@richrobe Could you have a first look regrading general structure and scope? Also can you have a look at the sleep analysis section and add citations for that?

Todos:

• [x] Add image for dataset/algo/pipeline (same as in general docs)
• [x] Fix missing dois

It might be helpful to have access to the actual datapoints in the custom aggregator.

This could be used to weight results (e.g. by the number of events in each dp, or by the type of "task" represented by each datapoint.

Parameters with leading "_" can lead to problems because of the way how Python splits strings based on a token in reverse mode:

However, I would suggest to just not allow leading "_" for parameters. That leads to strange names (with dripple-underscores) anyway

At the moment all attributes with a trailing underscore are considered results. This works nice in a general case.

It becomes a little annoying when using dataclasses/attrs, as we need to exclude these fields explicitly, when we provided type information for them.

Further, to don't make the API even more confusing, we only allow the run method to write these attributes. The optimize methods have no way to write results (see #46, #45).

Overcomming both issues, we could allow to define nested typed classes to store results in. With a little bit of magic (see below), we can enforce that each instance automatically gets a fresh instance of the class to write results in.

We could either enforce naiming of the these objects (e.g. r_, R_) for the actual results or we could use the decorator to mark and find them.

With that we could also allow to define specific additional optimization results. Either using a specific name or a seperate decorator.

from typing import Any, Literal, Optional, Type, Generic, TypeVar, Union, overload

T = TypeVar("T")


class _Generic:
    def __setattr__(self, __name: str, __value: Any) -> None:
        super().__setattr__(__name, __value)

    def __getattr__(self, __name: str) -> None:
        return self.__dict__[__name]


class result(Generic[T]):
    name: str
    klass: Type[T]

    def __init__(self, klass: Type[T] = _Generic):
        self.klass = klass

    def __set_name__(self, owner, name):
        self.name = name

    @overload
    def __get__(self, obj: Literal[None], objtype: Any) -> Type[T]:
        pass

    @overload
    def __get__(self, obj: object, objtype: Any) -> T:
        pass

    def __get__(self, obj: Optional[object], objtype: Any = None) -> Union[T, Type[T]]:
        if obj:
            new_instance = self.klass()
            name: str = getattr(self, "name", self.klass.__name__)
            setattr(obj, name, new_instance)
            return new_instance
        return self.klass


class ParentClassWithTypedResults:
    @result
    class R_:
        test: str


class ParentClassUntypedResults:
    r_ = result()


a = ParentClassUntypedResults()
a.something = "bla"

a = ParentClassWithTypedResults()
a.test = "blas"

Unfortunately, Pycharms autocomplete is not happy with the decorator and we don't get typechecking or autocomplete for the TypedResults case. VsCode works

#13 implements a basic version of a representation.

However, it has some remaining issues:

• [ ] Long representations are alls hown in the same line (maybe use this: https://github.com/scikit-learn/scikit-learn/blob/056f993b411c1fa5cf6a2ced8e51de03617b25b4/sklearn/base.py#L104)
• [ ] objects with unuasual representaitons (e.g. pd.DataFrames) can mess up the formatting, when included as a paramter)

Some further notes:

• sklearn does something very elegant and does not include parameters in the repr that are still at their default value.
• Dataset already has a repr that does not include the parameters. It might be nice to include the custom user parameters in that representation as well.