What it is

My pull request does: This change will try to compress the flat list of hyperparams, into a more structured representation. Please refer the following issue #486 for more information. I haven't added any tests for now, i wanted to make sure this implementation is reviewed.

How it works

I coded it this way: Current implementation will try to read the all_hps = pipeline.get_hyperparams() output and try to compress this particular output into more structured and readable format.

Example usage

Here is how you can use this new code as a end user:

Note: 
Please make dimensions and types clear to the reader.
E.g.: in the event fictious data is processed in this code example.
>>>all_hps = pipeline.get_hyperparams()\
>>>all_hps_shortened = all_hps.compress()
>>>print(type(all_hps_shortened))
...<class 'neuraxle.hyperparams.space.CompressedHyperparameterSamples'>
>>>pprint(all_hps_shortened)
...[
    {

        "step_name": "step1",
        "hyperparams": {'copy': True, 'iterated_power': 'auto'},
        "ancestor_steps": ["root"]
    },
    {

        "step_name": "step2",
        "hyperparams": {'copy': True, 'iterated_power': 'auto'},
        "ancestor_steps": ["root"]
    }

]

Raw data is a mixed of text columns, numerical columns, date columns, category (string based) columns.

Pre-processed data is numerical data with : 1Hot encoding of text columns 1Hot encoding of category data. 1Hot encoding of numerical (binning).

1. The pipeline should manage enlargement of initial columns, and different types of pre-processing based on different date types.

2. Keeps track of column name changes is useful (but complicated to handlle).

Key of AutoML is mostly the pre-processing part.

I think it would be a neat addition to generate Discrete-value hyperparams. You could redefine Boolean (which should really be called Bernouilli in my opinion) to be Binomial(1,p).

In the mean time it is possible to use Choice with custom probabilities to model this distribution.

Edit : Side question, lets say that, for my random hyperparameter search, I'd really like to use learning_rate = 10**uniform(-2,-4). That doesn't seem to be an option within the current state of the framework; I'd have to create my own class ExpUniform (which maybe could be already included). Would there be a not too complex way to introduce operator on these distributions instead? Or even if a complex option is on the table, are all attempts at doing this fundamentally overkill and computationally wasteful for the conceptual scope of the distributions in this file?

What it is

Changes in Mixin

• Adds a private method, called at initialisation, in ForceHandleMixin to ensure the basic functions _X_data_container inherited from _TransformerStep and _FittableStep are redefined. Previously, failure to overrides theses methods led to infinite recursion.
• Defines MixinForBaseTransformer which all other Mixin inherits. This Mixin asserts, at initilization, that BaseTransform is inherited from the instance and that it has been initialized. This change has the purpose to ensure that the initialization of base class is done in proper order (i.e. Steps before Mixin). (fixes #412)
• NonTransformableMixin defines a _transform_data_container method.

Changes in AutoML class

• Defined _transform_data_container and _fit_transfrom_data_contain to launch a NotImplementError (instead of executing an infinite recurrence)
• Fixed the undefined local variable error which would happen when a non-terminal exception that happened during the first trial would be catch by the try-catch in the main loop
• Added a flag in AutoML's contructor to define whether its should raise all error that happen during trial or only the one previously defined (EOFError, KeyboardInterrupt, SystemError, SystemExit). This is useful for moments we want all error to be raised (such as unittests)

Overhaul of the service assertion mechanism

There is now two different way to make service assertion; both of which are applicable on any step.

• SomeStep().assert_has_service_at_execution() wrap the SomeStep() instance with a LocalServiceAssertionWrapper and will test the presence of the service right before SomeStep's execution in the will_process call of LocalServiceAssertionWrapper
• SomeStep().assert_has_service() wrap SomeStep() with the GlobalyRetrievableWrapper which also tests the presence of the service at execution (like the LocalServiceAssertionWrapper)
• GlobalServiceAssertionExecutorMixin allows retrieval of all GlobalyRetrievableWrapper instances within the pipeline and tests the presence of required services.
• StepWithContext nows inherits of GlobalServiceAssertionExecutorMixin and thus test services at the root of the pipeline (i.e. beginning of pipeline execution)

Reason for this change : The previous version, which only asserted presence of service in a StepWithContext's will_process instance, did not allow to assert presence of a service which is registered during the execution of a pipeline.

Other minor changes

• _HasSavers now has an add_saver function
• Fixed the initialization order of TruncableSteps (fixes #422, mentionned in #369)
• Identity now inherits from NonFittableMixin and BaseStep instead of BaseTransformer
• test_could_have_context.py is now test_service_assertions.py (and has been extended)
• Removed function MetaStepMixin._ensure_proper_mixin_init_order since its now redundant with the MixinForBaseTransformer interface.
• examples/getting_started/plot_non_fittable_mixin.py has been adjusted to deal with the ForceHandleMixin changes
• fixes #403
• fixes #404

Hi @alexbrillant @Eric2Hamel,

We'd need a way to have a validation curve during training to be able to do this:

• Early stopping in deep learning mdoels when validation loss starts to raise
• Being able to plot train/val curves to debug the model's training and convergence
• Being able to send train/val curves to the AutoML algorithm (as model features that can be getted by model.introspect() after a training), so that the AutoML algorithm can also feel things as we see them when in look at those charts.

How should we change the API of Neuraxle in the next big release, 0.3.0? Neuraxle is currently at 0.2.1. The ability to plot the validation curve is something we procrastinated for too long.

Suggestion: the MiniBatchSequentialPipeline could be responsible for doing this by calling .fit on train, then .predict on the validation data, then train again... and so forth. Any other ideas? I also thought of:

• Doing the train/validation split at each mini-batch, inside the fit of the model, but this is very bad.
• Adding 2 more arguments to fit, but this is bad.
• Adding 1 more argument to transform, but this is bad and we should be using predict instead.

So the only solution to me looks like making the MiniBatchSequentialPipeline class manage that. However, how should a user tell the MiniBatchSequentialPipeline what is validation data? Would the MiniBatchSequentialPipeline instead just need a ValidationSplitWrapper (#174) and instead let the wrapper do the alternation of calling fit then predict then fit then predict again?

Thank you and Best Regards, Guillaume

Describe the bug When training AutoML on ResumablePipeline, logs are full of messages like these: UserWarning: Cannot Load Step /models/resumable_pipeline/AutoML/ResumablePipeline (ResumablePipeline:ResumablePipeline) With Step Saver JoblibStepSaver. saver.class.name))

To Reproduce

  HP_real = HyperparameterSpace({
    "learning_rate": Uniform(1e-5, 1),
    "max_depth": RandInt(2, 4),
    "n_estimators": Choice([30,60,90,100,130])
})

    pipeline_sk = ResumablePipeline([  
    # A Pipeline is composed of multiple chained steps. Steps
    # can alter the data before passing it to the next steps.
    AddFeatures([
        PCA(n_components=2),
        FastICA(n_components=2),
    ]),
    RidgeModelStacking([
        RandomForestRegressor(),
        GradientBoostingRegressor(warm_start=False, min_samples_leaf=2, random_state=42)  # validation_fraction = 0.2
    ])
], cache_folder=resumable_pipeline_folder).set_hyperparams_space(HP_real)

time_a = time.time()
auto_ml = AutoML(
    pipeline_sk,
    #AutoMLContainer  = AutoMLContainer(main_scoring_metric_name = "mse"),
    refit_trial=True,
    n_trials=int(epochs),
    cache_folder_when_no_handle=resumable_pipeline_folder,
    validation_splitter=ValidationSplitter(test_set_ratio),
    hyperparams_optimizer=RandomSearchHyperparameterSelectionStrategy(),
    scoring_callback=ScoringCallback(mean_squared_error, higher_score_is_better=False),
    callbacks=[
        MetricCallback('mse', metric_function=mean_squared_error, higher_score_is_better=False)
    ]
)
auto_ml = auto_ml.fit(train, y_train)  # if you use custom label encoder, then fit takes in the whole data at a time

Expected behavior I am expecting that there would be no errors such as these. The goal is to be able to warm start training from old checkpoints. Additional context Add any other context about the problem here.

Many classes have incomplete docstring. Sometimes, there are also typos.

The docstrings in classes are used with the sphinx documentation builder to build the website's complete documentation API: https://www.neuraxle.neuraxio.com/stable/api.html

Step Saving Checkpoints Are 100% valid now !! 🚀

• Moved all resume, and should resume for steps with children inside _HasChildrenMixin
• Simplified ResumablePipeline algo ALOT
• Fixed step saving checkpoints : step saving by summary id !
• Added summary id to save, and load (this is a breaking change :/ we could maybe do something to avoid it.... like method signatures ?) or we could live with it...

WORK IN PROGRESS TODO: docstrings validate/avoid breaking change weird merge some typings broke

This needed to be done... I felt really inspired and did this real fast tonight lol If we want to ship properly, we need to have this done because it is written in the readme...

Can you add an example in the documentation for:

neuraxle.steps.output_handlers.InputAndOutputTransformerMixin
neuraxle.steps.output_handlers.OutputTransformerWrapper

Especially on how to access the X and Y data and return them.

In my use case I want to set Y based on X values (financial forecasting)

thx!

What it is

My pull request does: This change will try to compress the flat list of hyperparams, into a more structured representation. Please refer the following PR for more information. I haven't added any tests for now, i wanted to make sure this implementation is reviewed.

How it works

I coded it this way: Current implementation will try to read the all_hps = pipeline.get_hyperparams() output and try to compress this particular output into more structured and readable format.

Example usage

Here is how you can use this new code as a end user:

Note: 
Please make dimensions and types clear to the reader.
E.g.: in the event fictious data is processed in this code example.

>>>all_hps = pipeline.get_hyperparams()\
>>>all_hps_shortened = all_hps.compress()
>>>print(type(all_hps_shortened))
...<class 'neuraxle.hyperparams.space.CompressedHyperparameterSamples'>
>>>pprint(all_hps_shortened)
...[
    {

        "step_name": "step1",
        "hyper_parameters": OrderedDict([('copy', True), ('iterated_power', 'auto')]),
        "ancestor_steps": ["root"]
    },
    {

        "step_name": "step1",
        "hyper_parameters":OrderedDict([('copy', True), ('iterated_power', 'auto')]),
        "ancestor_steps": ["root"]
    }

]

Is your feature request related to a problem? Please describe. Hyperparam names are too long in nested steps

Describe the solution you'd like A way to compress the names so as to make them shorter. More specifically, I think that an automated algorithm for all existing ML pipelines could be built. That would be to do something like:

all_hps = pipeline.get_hyperparams()
all_hps_shortened = all_hps.compress()
pprint(all_hps_shortened)

Then we'd see something like this in the pprint:

{
    "*__MetaStep__*__SKLearnWrapper_LinearRegression__C": 1000,
    "*__SomeStep__hyperparam3": value,
    "*__SKLearnWrapper_BoostedTrees__count": 10
}

That is, the unique paths to some steps were compressed using the star (*) operator. The Star operator means "one or more steps between". But the way the paths are compressed would be lossless, in the sense that the original names could ALWAYS be retrieved given the original pipeline's tree structure.

Describe alternatives you've considered Using custom ways to flush words and compress them. That seems good, but it doesn't seem to generalize to all pipelines that could exist.

Additional context Hyperparameter names were said to be too long as well in #478

Additional idea For hyperparameters, given the fact that in the future every model may need to name its expected hyperparams, then it may be possible to use their name only and directly if there is no other step with the same hyperparams. If another step uses the same hyperparam names, then compression with the "*" could go up in the tree to find the first non-common parent name or something.

More ideas are needed to be sure we do this the right way.

Describe the bug

Currently the testing_neuraxle directory is missing from the sdist on PyPI.

To Reproduce

curl -LO https://pypi.io/packages/source/n/neuraxle/neuraxle-0.8.1.tar.gz
tar xzf neuraxle-0.8.1.tar.gz
ls -1 neuraxle-0.8.1/testing_neuraxle

Expected behavior

Ideally the tests would be included in the sdist to make it easier to test after building/installing the package.

Suggested Fix

One solution would be to include a MANIFEST.in and add...

recursive-include testing_neuraxle *.py

...as well as anything else that might be needed to run the tests (like other directories or other extension types).

Additional context

This came up in trying to build & test a Conda package for conda-forge.

xref: https://github.com/conda-forge/staged-recipes/pull/16566#issuecomment-1217118836

cc @guillaume-chevalier (for awareness)

The problem: Currently the neuraxle BaseStep has a fit method signature with only 2 parameters (data_inputs, expected_outputs). In libraries like keras it is possible to have additional arguments being passed to the fit method. This could be things like validation generators if the main data_inputs is a data generator as well.

This means if we want to wrap a keras model that takes two data generators, in a subclass of BaseStep, then it wouldnt be a straight forward implementation.

Solution: It would be extremely useful if an additional **kwargs is added to the base step fit method(in one or more of the Mixin classes) to enable passing arbitrary arguments to the custom estimator implementations.

Describe the bug From version 0.7.0 and later, these callbacks don't have access to the pipeline anymore since the pipeline isn't stored anymore in the TrialSplit, but just in the AutoML class.

To Reproduce Update to 0.7.0 and run these tests that are skipped.

Expected behavior Callbacks are supposed to log models using the repo.

Suggested Fix Possibly have the TrialSplits contain their trained model again, instead of letting the Trainer call the model.

Additional context 0.7.0 is not merged nor deployed at the moment of writing this issue.

There is currently no ParallelColumnTransformer, and lack of tests for ParallelFeatureUnion.

Suggested fix:

• Parametrize FeatureUnion tests and ColumnTransformer tests.

Is your feature request related to a problem? Please describe. As the context acts like a stack of function calls with each level having its own "variables"-like saving in the context (e.g.: Memory (as per #443 ), it might be good that if adding a service at the middle of the pipeline, that the service is scoped and local to just this context.pushed level (like a programming language stack / Assembly CPU Stack that is cleared upon stack push).

Describe the solution you'd like When pushing and popping to context, use a recursive context data structure instead of lists of parents and a single service dict. This was done originally in the first releases of Neuraxle ever, but that was change for some reasons I don't know. It seems that the original intuition was possibly better.

Describe alternatives you've considered Singly-linked list (stack data structure / design pattern) for the context's parents, and services. Could set global v.s. local services as well (like programming languages' global v.s. local scopes when declaring/using variables).

Additional context None

Is your feature request related to a problem? Please describe. Meta-describing a pipeline from, say, a loaded configuration json would require to compose the objects back together. To do so, from object names, we'd need an object store to save the objects to upon creating them, including custom objects.

Describe the solution you'd like BaseStep to register objects to the global store using a descriptor.

Describe alternatives you've considered The "Pattern: language integrated registration" of the article found below as well. But it adds too much boilerplate.

Additional context See this article, especially the sections:

• Pattern: metaclass based registration
• Pattern: DSL-based declaration

Metaclass-based registration could be used as well for specifying hyperparameter distributions and more objects. This could be like Orion strings in the Orion framework.