Elliot is a comprehensive recommendation framework that analyzes the recommendation problem from the researcher's perspective.

Last update: Nov 29, 2022

Overview

ELLIOT

Elliot is a comprehensive recommendation framework that analyzes the recommendation problem from the researcher's perspective. It conducts a whole experiment, from dataset loading to results gathering. The core idea is to feed the system with a simple and straightforward configuration file that drives the framework through the experimental setting choices. Elliot untangles the complexity of combining splitting strategies, hyperparameter model optimization, model training, and the generation of reports of the experimental results.

The framework loads, filters, and splits the data considering a vast set of strategies (splitting methods and filtering approaches, from temporal training-test splitting to nested K-folds Cross-Validation). Elliot optimizes hyperparameters for several recommendation algorithms, selects the best models, compares them with the baselines providing intra-model statistics, computes metrics spanning from accuracy to beyond-accuracy, bias, and fairness, and conducts statistical analysis (Wilcoxon and Paired t-test).

Elliot aims to keep the entire experiment reproducible and put the user in control of the framework.

Installation

Elliot works with the following operating systems:

Linux
Windows 10
macOS X

Elliot requires Python version 3.6 or later.

Elliot requires tensorflow version 2.3.2 or later. If you want to use Elliot with GPU, please ensure that CUDA or cudatoolkit version is 7.6 or later. This requires NVIDIA driver version >= 10.1 (for Linux and Windows10).

Please refer to this document for further working configurations.

Install from source

CONDA

git clone https://github.com//sisinflab/elliot.git && cd elliot
conda create --name elliot_env python=3.8
conda activate elliot_env
pip install --upgrade pip
pip install -e . --verbose

VIRTUALENV

git clone https://github.com//sisinflab/elliot.git && cd elliot
virtualenv -p /usr/bin/python3.6 venv # your python location and version
source venv/bin/activate
pip install --upgrade pip
pip install -e . --verbose

Quick Start

Elliot's entry point is the function run_experiment, which accepts a configuration file that drives the whole experiment. In the following, a sample configuration file is shown to demonstrate how a sample and explicit structure can generate a rigorous experiment.

from elliot.run import run_experiment

run_experiment("configuration/file/path")

The following file is a simple configuration for an experimental setup. It contains all the instructions to get the MovieLens-1M catalog from a specific path and perform a train test split in a random sample way with a ratio of 20%.

This experiment provides a hyperparameter optimization with a grid search strategy for an Item-KNN model. Indeed, it is seen that the possible values of neighbors are closed in squared brackets. It indicates that two different models equipped with two different neighbors' values will be trained and compared to select the best configuration. Moreover, this configuration obliges Elliot to save the recommendation lists with at most 10 items per user as suggest by top_k property.

In this basic experiment, only a simple metric is considered in the final evaluation study. The candidate metric is nDCG for a cutoff equal to top_k, unless otherwise noted.

experiment:
  dataset: movielens_1m
  data_config:
    strategy: dataset
    dataset_path: ../data/movielens_1m/dataset.tsv
  splitting:
    test_splitting:
      strategy: random_subsampling
      test_ratio: 0.2
  models:
    ItemKNN:
      meta:
        hyper_opt_alg: grid
        save_recs: True
      neighbors: [50, 100]
      similarity: cosine
  evaluation:
    simple_metrics: [nDCG]
  top_k: 10

If you want to explore a basic configuration, and an advanced configuration, please refer to:

basic_configuration

advanced_configuration

You can find the full description of the two experiments in the paper.

Contributing

There are many ways to contribute to Elliot! You can contribute code, make improvements to the documentation, report or investigate bugs and issues

We welcome all contributions from bug fixes to new features and extensions.

Feel free to share with us your custom configuration files. We are creating a vault of reproducible experiments, and we would be glad of mentioning your contribution.

Reference Elliot in your blogs, papers, and articles.

Talk about Elliot on social media with the hashtag #elliotrs.

Cite

If you find Elliot useful for your research or development, please cite the following paper:


@article{DBLP:journals/corr/abs-2103-02590,
  author    = {Vito Walter Anelli and
               Alejandro Bellog{\'{\i}}n and
               Antonio Ferrara and
               Daniele Malitesta and
               Felice Antonio Merra and
               Claudio Pomo and
               Francesco M. Donini and
               Tommaso Di Noia},
  title     = {Elliot: a Comprehensive and Rigorous Framework for Reproducible Recommender
               Systems Evaluation},
  journal   = {CoRR},
  volume    = {abs/2103.02590},
  year      = {2021}
}

The Team

Elliot is developed by

Vito Walter Anelli^* ([email protected])
Alejandro Bellogín ([email protected])
Antonio Ferrara ([email protected])
Daniele Malitesta ([email protected])
Felice Antonio Merra ([email protected])
Claudio Pomo^* ([email protected])
Francesco Maria Donini ([email protected])
Tommaso Di Noia ([email protected])

It is maintained by SisInfLab Group and Information Retrieval Group.

^* Corresponding authors

License

ELLIOT uses APACHE2 License.

Acknowledgements

SliM and an alternative KNN-CF implementation refer to RecSys2019_DeepLearning_Evaluation

Comments

Dict Comprehension speeds up data preparation

Describe the bug I have been using datasets to make experiments with elliot where userId and itemId are strings. In these cases I have noticed speed improvement if I use dict comprehension instead of loops for the preparation of the dataset.

To Reproduce Link for the commit on local branch. I could create a pull request if you think that it would be an improvement on your side as well.

import pandas as pd
import time

data = pd.read_csv('data/cat_dbpedia_movielens_1m/dataset.tsv',sep='\t',header=None)
data.columns = ['userId','itemId','rating','timestamp']
#change type of column 
data.userId = data.userId.astype(str)

##Loop 1 vs dict comprehension
t0 = time.time()
users = list(data['userId'].unique())
ratings = {}
for u in users:
    sel_ = data[data['userId'] == u]
    ratings[u] = dict(zip(sel_['itemId'], sel_['rating']))
t1 = time.time()
print(t1-t0) #356.07s

t0 = time.time()
ratings = {k: f.groupby('itemId')['rating'].apply(float).to_dict() for k, f in data.groupby('userId')}
t1 = time.time()
print(t1-t0) #29.83

##Loop 2 vs dict comprehension
dataframe = data 
#users are not included in the time loop since they are provided as parameter in the method
users = list(data['userId'].unique())
t0 = time.time()
ratings = {}
for u in users:
    sel_ = dataframe[dataframe['userId'] == u]
    ratings[u] = dict(zip(sel_['itemId'], sel_['rating']))
t1 = time.time()
print(t1-t0) #371.22

t0 = time.time()
ratings = {k: f.groupby('itemId')['rating'].apply(float).to_dict() for k, f in dataframe.groupby('userId') if k in users}
t1 = time.time()
print(t1-t0) #32.44s

Expected behavior I would expect the time required for data preparation to be on the same level regardless of the type of userId/itemId.

Screenshots

System details (please complete the following information):

OS: Ubuntu 20.04
Python: 3.8.10

Additional context

enhancement

opened by pkaram 3

$No such file or directory: 'D:\\develop\\elliot\\data\\movielens_1m\\dataset.tsv'$

No such file or directory: 'D:\\develop\\elliot\\data\\movielens_1m\\dataset.tsv'

Version Number: 0.3.1 2022-10-14 15:42:23.398801: I Start experiment Traceback (most recent call last): 2022-10-14 15:42:23.415288: I There will be the splitting File ".\start_experiments.py", line 8, in run_experiment(f"config_files/{args.config}.yml") File "D:\develop\elliot\elliot\run.py", line 60, in run_experiment dataloader = dataloader_class(config=base.base_namespace) File "D:\develop\elliot\elliot\dataset\dataset.py", line 95, in init self.dataframe = pd.read_csv(path_dataset, sep="\t", header=None, names=self.column_names) File "D:\Users\Anaconda3\envs\elliot_env\lib\site-packages\pandas\io\parsers.py", line 688, in read_csv return _read(filepath_or_buffer, kwds) File "D:\Users\Anaconda3\envs\elliot_env\lib\site-packages\pandas\io\parsers.py", line 454, in _read parser = TextFileReader(fp_or_buf, **kwds) File "D:\Users\Anaconda3\envs\elliot_env\lib\site-packages\pandas\io\parsers.py", line 948, in init self._make_engine(self.engine) File "D:\Users\Anaconda3\envs\elliot_env\lib\site-packages\pandas\io\parsers.py", line 1180, in _make_engine self._engine = CParserWrapper(self.f, **self.options) File "D:\Users\Anaconda3\envs\elliot_env\lib\site-packages\pandas\io\parsers.py", line 2010, in init self._reader = parsers.TextReader(src, **kwds) File "pandas_libs\parsers.pyx", line 382, in pandas._libs.parsers.TextReader.cinit File "pandas_libs\parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source FileNotFoundError: [Errno 2] No such file or directory: 'D:\develop\elliot\data\movielens_1m\dataset.tsv'

opened by imitn 1
Basic Configuration example has different paths for dataset.tsv
Describe the bug Running sample basic (https://github.com/sisinflab/elliot/blob/master/sample_basic.py) encounters a problem for file not found. This is because the dataset.tsv is downloaded into data/movilens_1m folder but the configuration file loaded by the same script (https://github.com/sisinflab/elliot/blob/master/config_files/basic_configuration.yml) looks for the dataset in /data/cat_dbpedia_movielens_1m.

To Reproduce Run sample_basic.py

Expected behavior dataset.tsv should be downloaded into a folder reachable by the code

Screenshots

System details (please complete the following information):

OS: [e.g. iOS]

Python Version [e.g. 3.6.9]

Version of the Libraries [e.g. 22]

Additional context Add any other context about the problem here.
opened by karapostK 1
Suggestion: SIGIR 2021 E-Commerce Workshop Data Challenge

Hi! I just discovered this project from links in a discussion area from another project. Having a look through here, I appreciate your features in the area of splitting, unique metrics, and stats tests!

I learned the other day that the SIGIR 2021 Conference has an e-commerce workshop data challenge, called In-session prediction for purchase intent and recommendations. They just opened it up for participation.

https://sigir-ecom.github.io/data-task.html

After looking at elliot's input structure, it seems like it wouldn't be too far off to connect elliot to the data challenge to get some elliot benchmarks out. Though, I didn't dig into the available models enough to know how they map to in-session datasets.

Unfortunately I am unable to give it a go due to other professional commitments. But I figured I'd share it here as well as an opportunity to grow awareness around elliot, in the event others are interested.
enhancement

opened by tszumowski 1
Filenaming problem in Windows 10

Hello,

Following the instructions provided in the documentation, I have succesfully installed the Elliot library using Conda v.4.8.2. and Python 3.7.10 on a Windows 10 system.

However, I have found some problems at the moment of executing some experiments on the framework. The library seems to work finely when I use an algorithm without hyperparameters (as it happens, for instance, with the MostPop algorithm). However, due to how the weight files are named, it does not allow me to execute algorithms with hyperparameters, such as UserKNN or ItemKNN.

At the moment, the project tries to create a directory or a file (I think is the first one), named algorithm_hyper1:val1_hyper2:val2_..._hyperN:valN where hyperX is the name of the X-th hyperparameter, valX is the value of such hyperparameter and N is the final number of hyperparameters.

However, Windows does not allow colons in the names of files/folders, so it stops, and returns an OSError as the one I show below:

Can anything be done to solve this?

To reproduce I share the configuration file that I used:

Thanks in advance.
bug

opened by JavierSanzCruza 1
FunkSVD finished abruptly

I have tried to run FunkSVD on MovieLens100k but it does finish the execution.

To Reproduce The screenshots of the running and config files used for reproducing what I did are as follows.

Expected behavior The screenshot of the execution trace is as follows.

I am Python 3.7 on a MacOS Big Sur version 11.1
help wanted question

opened by lbalby 1
Bump tensorflow from 2.3.2 to 2.4.0
Bumps tensorflow from 2.3.2 to 2.4.0.

Release notes

Sourced from tensorflow's releases.

TensorFlow 2.4.0

Release 2.4.0

Major Features and Improvements

tf.distribute introduces experimental support for asynchronous training of models via the tf.distribute.experimental.ParameterServerStrategy API. Please see the tutorial to learn more.

MultiWorkerMirroredStrategy is now a stable API and is no longer considered experimental. Some of the major improvements involve handling peer failure and many bug fixes. Please check out the detailed tutorial on Multi-worker training with Keras.

Introduces experimental support for a new module named tf.experimental.numpy which is a NumPy-compatible API for writing TF programs. See the detailed guide to learn more. Additional details below.

Adds Support for TensorFloat-32 on Ampere based GPUs. TensorFloat-32, or TF32 for short, is a math mode for NVIDIA Ampere based GPUs and is enabled by default.

A major refactoring of the internals of the Keras Functional API has been completed, that should improve the reliability, stability, and performance of constructing Functional models.

Keras mixed precision API tf.keras.mixed_precision is no longer experimental and allows the use of 16-bit floating point formats during training, improving performance by up to 3x on GPUs and 60% on TPUs. Please see below for additional details.

TensorFlow Profiler now supports profiling MultiWorkerMirroredStrategy and tracing multiple workers using the sampling mode API.

TFLite Profiler for Android is available. See the detailed guide to learn more.

TensorFlow pip packages are now built with CUDA11 and cuDNN 8.0.2.

Breaking Changes

TF Core:

Certain float32 ops run in lower precsion on Ampere based GPUs, including matmuls and convolutions, due to the use of TensorFloat-32. Specifically, inputs to such ops are rounded from 23 bits of precision to 10 bits of precision. This is unlikely to cause issues in practice for deep learning models. In some cases, TensorFloat-32 is also used for complex64 ops. TensorFloat-32 can be disabled by running tf.config.experimental.enable_tensor_float_32_execution(False).

The byte layout for string tensors across the C-API has been updated to match TF Core/C++; i.e., a contiguous array of tensorflow::tstring/TF_TStrings.

C-API functions TF_StringDecode, TF_StringEncode, and TF_StringEncodedSize are no longer relevant and have been removed; see core/platform/ctstring.h for string access/modification in C.

tensorflow.python, tensorflow.core and tensorflow.compiler modules are now hidden. These modules are not part of TensorFlow public API.

tf.raw_ops.Max and tf.raw_ops.Min no longer accept inputs of type tf.complex64 or tf.complex128, because the behavior of these ops is not well defined for complex types.

XLA:CPU and XLA:GPU devices are no longer registered by default. Use TF_XLA_FLAGS=--tf_xla_enable_xla_devices if you really need them, but this flag will eventually be removed in subsequent releases.

tf.keras:

The steps_per_execution argument in model.compile() is no longer experimental; if you were passing experimental_steps_per_execution, rename it to steps_per_execution in your code. This argument controls the number of batches to run during each tf.function call when calling model.fit(). Running multiple batches inside a single tf.function call can greatly improve performance on TPUs or small models with a large Python overhead.

A major refactoring of the internals of the Keras Functional API may affect code that is relying on certain internal details:

Code that uses isinstance(x, tf.Tensor) instead of tf.is_tensor when checking Keras symbolic inputs/outputs should switch to using tf.is_tensor.

Code that is overly dependent on the exact names attached to symbolic tensors (e.g. assumes there will be ":0" at the end of the inputs, treats names as unique identifiers instead of using tensor.ref(), etc.) may break.

Code that uses full path for get_concrete_function to trace Keras symbolic inputs directly should switch to building matching tf.TensorSpecs directly and tracing the TensorSpec objects.

Code that relies on the exact number and names of the op layers that TensorFlow operations were converted into may have changed.

Code that uses tf.map_fn/tf.cond/tf.while_loop/control flow as op layers and happens to work before TF 2.4. These will explicitly be unsupported now. Converting these ops to Functional API op layers was unreliable before TF 2.4, and prone to erroring incomprehensibly or being silently buggy.

Code that directly asserts on a Keras symbolic value in cases where ops like tf.rank used to return a static or symbolic value depending on if the input had a fully static shape or not. Now these ops always return symbolic values.

Code already susceptible to leaking tensors outside of graphs becomes slightly more likely to do so now.

Code that tries directly getting gradients with respect to symbolic Keras inputs/outputs. Use GradientTape on the actual Tensors passed to the already-constructed model instead.

Code that requires very tricky shape manipulation via converted op layers in order to work, where the Keras symbolic shape inference proves insufficient.

Code that tries manually walking a tf.keras.Model layer by layer and assumes layers only ever have one positional argument. This assumption doesn't hold true before TF 2.4 either, but is more likely to cause issues now.

... (truncated)

Changelog

Sourced from tensorflow's changelog.

Release 2.3.2

Bug Fixes and Other Changes

Fixes an access to unitialized memory in Eigen code (CVE-2020-26266)

Fixes a security vulnerability caused by lack of validation in tf.raw_ops.DataFormatVecPermute and tf.raw_ops.DataFormatDimMap (CVE-2020-26267)

Fixes a vulnerability caused by attempting to write to immutable memory region in tf.raw_ops.ImmutableConst (CVE-2020-26268

Fixes a CHECK-fail in LSTM with zero-length input (CVE-2020-26270)

Fixes a security vulnerability caused by accessing heap data outside of bounds when loading a specially crafted SavedModel (CVE-2020-26271)

Solves an OOM issue on TPUs when XLA contexts use fused average updates

Updates libjpeg-turbo to 2.0.5 to handle CVE-2020-13790.

Updates junit to 4.13.1 to handle CVE-2020-15250.

Updates PCRE to 8.44 to handle CVE-2019-20838 and CVE-2020-14155.

Updates sqlite3 to 3.44.0 to keep in sync with master branch.

Release 2.2.2

Bug Fixes and Other Changes

Fixes an access to unitialized memory in Eigen code (CVE-2020-26266)

Fixes a security vulnerability caused by lack of validation in tf.raw_ops.DataFormatVecPermute and tf.raw_ops.DataFormatDimMap (CVE-2020-26267)

Fixes a vulnerability caused by attempting to write to immutable memory region in tf.raw_ops.ImmutableConst (CVE-2020-26268

Fixes a CHECK-fail in LSTM with zero-length input (CVE-2020-26270)

Fixes a security vulnerability caused by accessing heap data outside of bounds when loading a specially crafted SavedModel (CVE-2020-26271)

Prevents memory leaks in loading SavedModels that import functions

Updates libjpeg-turbo to 2.0.5 to handle CVE-2020-13790.

Updates junit to 4.13.1 to handle CVE-2020-15250.

Updates PCRE to 8.44 to handle CVE-2019-20838

... (truncated)

Commits

582c8d2 Merge pull request #44220 from tensorflow-jenkins/relnotes-2.4.0rc0-18048

c16387f Update RELEASE.md

4cf406c Update RELEASE.md

3f35ef2 Update RELEASE.md

3647e8e Update RELEASE.md

281c7d5 Update RELEASE.md

91ec75f Update RELEASE.md

ed5ad82 Update RELEASE.md

1267bba Update RELEASE.md

13a4067 Update RELEASE.md

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 1
[Feature Request] Early stopping?

Not seeing an option for early stopping based on validation metric performance. Running neural models with fixed numbers of epochs is either time-consuming (treating epoch number as a hyperparameter) or risks under/overfitting (fixing the epochs).
enhancement

opened by deklanw 0
Fix setup issues

Fixes https://github.com/sisinflab/elliot/issues/2

In the latest pip parse_requirements requires a session parameter, and, imports have been shuffled around.

Also, in the README instructions, activate the newly created conda environment

Just a note, I imitated the way the code already worked, but relying on the pip internals to do this may not be the best idea https://github.com/pypa/pip/issues/2422

opened by deklanw 0

Setup doesn't work with latest pip

    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/deklan/code/forks/elliot/setup.py", line 30, in <module>
        install_reqs = parse_requirements('requirements.txt')
    TypeError: parse_requirements() missing 1 required positional argument: 'session'

Pip version

pip 21.0.1

opened by deklanw 0

Evaluration Module: Diversity - Gini Index

Describe the bug According to the algorithm, as it is designed in Elliot, Gini Index can take values > 1, however, according to the bibliography, this is not possible. Could you explain further how is that possible in your algorithm?

opened by nikosT 0

Issues installing on M1 Mac

Please let me know if this has been already discussed before - I've tried finding any information and failed. I have attempted to install Elliot according to the instructions in the docs:

git clone https://github.com//sisinflab/elliot.git && cd elliot
virtualenv -p /usr/bin/pyhton3.6 venv # your python location and version
source venv/bin/activate
pip install --upgrade pip
pip install -e . --verbose

I am using a MacBook Pro with M1 Pro chip, 12.0.1 Monterey OS, Python version 3.9 installed with Homebrew. It looks like there is some issue with Cython and some installation quirk, but I really don't know how to approach solving this. Please let me know if anyone else had such an issue previously.

When I run the last pip install command I get the following terminal output:

Using pip 22.0.4 from /Users/anastasiia/Desktop/elliot/venv/lib/python3.9/site-packages/pip (python 3.9)
Looking in indexes: https://pypi.org/simple, https://anastasiia.klimashevskaia%40tv2.no:****@tv2norge.jfrog.io/tv2norge/api/pypi/ai-python-local/simple
Obtaining file:///Users/anastasiia/Desktop/elliot
  Running command python setup.py egg_info
  running egg_info
  creating /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info
  writing /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/dependency_links.txt
  writing requirements to /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/requires.txt
  writing top-level names to /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/top_level.txt
  writing manifest file '/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/SOURCES.txt'
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-pip-egg-info-su39l3jq/elliot.egg-info/SOURCES.txt'
  Preparing metadata (setup.py) ... done
Collecting hyperopt==0.2.5
  Using cached hyperopt-0.2.5-py2.py3-none-any.whl (965 kB)
Collecting numpy==1.18.0
  Using cached numpy-1.18.0.zip (5.4 MB)
  Running command pip subprocess to install build dependencies
  Looking in indexes: https://pypi.org/simple, https://anastasiia.klimashevskaia%40tv2.no:****@tv2norge.jfrog.io/tv2norge/api/pypi/ai-python-local/simple, https://anastasiia.klimashevskaia%40tv2.no:****@tv2norge.jfrog.io/tv2norge/api/pypi/ai-python-local/simple
  Collecting setuptools
    Using cached setuptools-62.1.0-py3-none-any.whl (1.1 MB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting Cython>=0.29.13
    Using cached Cython-0.29.28-py2.py3-none-any.whl (983 kB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-0.29.28 setuptools-62.1.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Running from numpy source directory.
  running egg_info
  creating numpy.egg-info
  writing numpy.egg-info/PKG-INFO
  writing dependency_links to numpy.egg-info/dependency_links.txt
  writing entry points to numpy.egg-info/entry_points.txt
  writing top-level names to numpy.egg-info/top_level.txt
  writing manifest file 'numpy.egg-info/SOURCES.txt'
  package init file 'numpy/distutils/mingw/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/include/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/include/numpy/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/include/numpy/random/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/tests/data/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/npysort/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/multiarray/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/npymath/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/common/__init__.py' not found (or not a regular file)
  package init file 'numpy/core/src/umath/__init__.py' not found (or not a regular file)
  package init file 'numpy/linalg/lapack_lite/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/src/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/mixed/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/array_from_pyobj/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/kind/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/common/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/assumed_shape/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/parameter/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/string/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/size/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/tests/src/regression/__init__.py' not found (or not a regular file)
  package init file 'numpy/f2py/src/test/__init__.py' not found (or not a regular file)
  package init file 'numpy/lib/tests/data/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/include/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/_examples/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/_examples/cffi/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/_examples/cython/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/_examples/numba/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/mt19937/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/sfc64/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/pcg64/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/legacy/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/philox/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/splitmix64/__init__.py' not found (or not a regular file)
  package init file 'numpy/random/src/distributions/__init__.py' not found (or not a regular file)
  package init file 'numpy/_build_utils/src/__init__.py' not found (or not a regular file)
  reading manifest file 'numpy.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  no previously-included directories found matching 'doc/build'
  no previously-included directories found matching 'doc/source/generated'
  no previously-included directories found matching 'benchmarks/env'
  no previously-included directories found matching 'benchmarks/results'
  no previously-included directories found matching 'benchmarks/html'
  no previously-included directories found matching 'benchmarks/numpy'
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '*.pyo' found anywhere in distribution
  warning: no previously-included files matching '*.pyd' found anywhere in distribution
  warning: no previously-included files matching '*.swp' found anywhere in distribution
  warning: no previously-included files matching '*.bak' found anywhere in distribution
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files found matching 'LICENSES_bundled.txt'
  adding license file 'LICENSE.txt'
  writing manifest file 'numpy.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  Running from numpy source directory.
  setup.py:425: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
    run_build = parse_setuppy_commands()
  /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/tools/cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    required_version = LooseVersion('0.29.13')
  /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/tools/cythonize.py:77: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    if LooseVersion(cython_version) < required_version:
  Processing numpy/random/_bounded_integers.pxd.in
  Processing numpy/random/_philox.pyx
  Processing numpy/random/_bit_generator.pyx
  Processing numpy/random/_bounded_integers.pyx.in
  Processing numpy/random/_sfc64.pyx
  Processing numpy/random/_mt19937.pyx
  Processing numpy/random/mtrand.pyx
  Processing numpy/random/_generator.pyx
  Processing numpy/random/_pcg64.pyx
  Processing numpy/random/_common.pyx
  Cythonizing sources
  blas_opt_info:
  blas_mkl_info:
  customize UnixCCompiler
    libraries mkl_rt not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  blis_info:
    libraries blis not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  openblas_info:
    libraries openblas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  atlas_3_10_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries tatlas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  atlas_3_10_blas_info:
    libraries satlas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  atlas_blas_threads_info:
  Setting PTATLAS=ATLAS
    libraries ptf77blas,ptcblas,atlas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  atlas_blas_info:
    libraries f77blas,cblas,atlas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  accelerate_info:
    libraries accelerate not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
  Library accelerate was not found. Ignoring
    libraries veclib not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
  Library veclib was not found. Ignoring
    FOUND:
      extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

    FOUND:
      extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  /bin/sh: svnversion: command not found
  non-existing path in 'numpy/distutils': 'site.cfg'
  lapack_opt_info:
  lapack_mkl_info:
    libraries mkl_rt not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  openblas_lapack_info:
    libraries openblas not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  openblas_clapack_info:
    libraries openblas,lapack not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  flame_info:
    libraries flame not found in ['/Users/anastasiia/Desktop/elliot/venv/lib', '/usr/local/lib', '/usr/lib']
    NOT AVAILABLE

  atlas_3_10_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries tatlas,tatlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries tatlas,tatlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries tatlas,tatlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
    NOT AVAILABLE

  atlas_3_10_info:
    libraries lapack_atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries satlas,satlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries satlas,satlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries satlas,satlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_3_10_info'>
    NOT AVAILABLE

  atlas_threads_info:
  Setting PTATLAS=ATLAS
    libraries lapack_atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries ptf77blas,ptcblas,atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_threads_info'>
    NOT AVAILABLE

  atlas_info:
    libraries lapack_atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries f77blas,cblas,atlas not found in /Users/anastasiia/Desktop/elliot/venv/lib
    libraries lapack_atlas not found in /usr/local/lib
    libraries f77blas,cblas,atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib
    libraries f77blas,cblas,atlas not found in /usr/lib
  <class 'numpy.distutils.system_info.atlas_info'>
    NOT AVAILABLE

    FOUND:
      extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers']
      extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
      define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py:275: UserWarning: Unknown distribution option: 'define_macros'
    warnings.warn(msg)
  running dist_info
  running build_src
  build_src
  building py_modules sources
  creating build
  creating build/src.macosx-12-arm64-3.9
  creating build/src.macosx-12-arm64-3.9/numpy
  creating build/src.macosx-12-arm64-3.9/numpy/distutils
  building library "npymath" sources
  Traceback (most recent call last):
    File "/Users/anastasiia/Desktop/elliot/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/anastasiia/Desktop/elliot/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/anastasiia/Desktop/elliot/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
      return hook(metadata_directory, config_settings)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
      self.run_setup()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 450, in <module>
      setup_package()
    File "setup.py", line 442, in setup_package
      setup(**metadata)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/core.py", line 171, in setup
      return old_setup(**new_attr)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
      return distutils.core.setup(**attrs)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
      super().run_command(command)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 35, in run
      egg_info.run()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/egg_info.py", line 26, in run
      self.run_command("build_src")
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1214, in run_command
      super().run_command(command)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/build_src.py", line 146, in run
      self.build_sources()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/build_src.py", line 157, in build_sources
      self.build_library_sources(*libname_info)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/build_src.py", line 290, in build_library_sources
      sources = self.generate_sources(sources, (lib_name, build_info))
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/build_src.py", line 380, in generate_sources
      source = func(extension, build_dir)
    File "numpy/core/setup.py", line 656, in get_mathlib_info
      st = config_cmd.try_link('int main(void) { return 0;}')
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-build-env-h1wqgh2x/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/config.py", line 241, in try_link
      self._check_compiler()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/command/config.py", line 82, in _check_compiler
      self.fcompiler = new_fcompiler(compiler=self.fcompiler,
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/__init__.py", line 884, in new_fcompiler
      compiler = get_default_fcompiler(plat, requiref90=requiref90,
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/__init__.py", line 855, in get_default_fcompiler
      compiler_type =  _find_existing_fcompiler(matching_compiler_types,
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/__init__.py", line 806, in _find_existing_fcompiler
      c.customize(dist)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/__init__.py", line 530, in customize
      linker_so_flags = self.flag_vars.linker_so
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/environment.py", line 40, in __getattr__
      return self._get_var(name, conf_desc)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/environment.py", line 56, in _get_var
      var = self._hook_handler(name, hook)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/__init__.py", line 709, in _environment_hook
      return hook()
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/gnu.py", line 353, in get_flags_linker_so
      flags = GnuFCompiler.get_flags_linker_so(self)
    File "/private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc/numpy/distutils/fcompiler/gnu.py", line 144, in get_flags_linker_so
      os.environ['MACOSX_DEPLOYMENT_TARGET'] = target
    File "/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 684, in __setitem__
      value = self.encodevalue(value)
    File "/opt/homebrew/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 756, in encode
      raise TypeError("str expected, not %s" % type(value).__name__)
  TypeError: str expected, not int
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/anastasiia/Desktop/elliot/venv/bin/python /Users/anastasiia/Desktop/elliot/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/tmp15w8xm0b
  cwd: /private/var/folders/gr/dpwnfzbj45sfzthq83r6cn7m0000gn/T/pip-install-1_bvhw4u/numpy_94acb307542d4878a85f46abd422e4fc
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

The output of my pip freeze:

absl-py==1.0.0
astunparse==1.6.3
cachetools==5.0.0
certifi==2021.10.8
charset-normalizer==2.0.12
flatbuffers==2.0
gast==0.5.3
google-auth==2.6.4
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.44.0
h5py==3.6.0
idna==3.3
importlib-metadata==4.11.3
keras==2.8.0
Keras-Preprocessing==1.1.2
libclang==13.0.0
Markdown==3.3.6
numpy==1.22.3
oauthlib==3.2.0
opt-einsum==3.3.0
protobuf==3.20.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
requests==2.27.1
requests-oauthlib==1.3.1
rsa==4.8
six==1.16.0
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-macos==2.8.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
typing_extensions==4.1.1
urllib3==1.26.9
Werkzeug==2.1.1
wrapt==1.14.0
zipp==3.8.0

Like I said, I would greatly appreciate any assistance. Thank you!

opened by anaklim 1

Simple metrics are producing different values for the same recommendations
Describe the bug Simple metrics ItemCoverage, UserCoverage, NumRetrieved are producing different values for the same recs/ folder between the procedure of calculating the model and evaluating it and the procedure of loading the model and evaluating it.

To Reproduce Steps to reproduce the behavior:

Run a model e.g. itemKNN which will produce the recommendations in the results/recs/ dir

Keep evaluation of the running model

Load the results/recs/ dir as a RecommendationFolder

Run evaluation only

Compare the two evaluations

In both cases, the input dataset has a strategy: fixed. The train.tsv and test.tsv files have previously been produced from a random 0.2 splitting procedure and used as they are for both cases.

System details (please complete the following information):

OS: Debian

Python Version 3.8

Version of the Libraries installed with conda elliot_env as described in doc
opened by nikosT 0
Sample Scripts Don't Run

I've tried installing the sir_elliot branch as well as the master branch of elliot but none of the existing sample scripts seem to work. I mostly got errors claiming there are no files or directories found. Please advise on how I should proceed or whether there are any alternative frameworks I should use as I need this for a project I'm working on.

opened by agb2k 0
Question

Hi,

Before going more in detail into the implementation of Elliot, I would like to know whether Elliot will be the right package for my experiments.

Basically, I am interested in context-aware recommender system (CARS) algorithms i.e., FM, FFM. When looking here I found some CARS algorithms but I don't see in the parameter how to add the side information (either user auxilliary information or item auxilliary information). (1) Would that mean that I only need to mention the extra side information in the YAML file only? Or does it mean that there are only specific algorithms implemented in Elliot that do take into account side information? If so could you list them please?

(2) Do you have somewhere an example of code running CARS algorithm and is it possible to reproduce your results?

Thank you in advance.

Looking forward to hearing back from you. Best, -- Manel.

opened by SlokomManel 0

Releases(v0.3.1)

v0.3.1(Jul 8, 2021)
ELLIOT v0.3.1 Release Notes

[v0.3.1] - 2021-07-05

Changed

fix AMF: insert of perturbation evaluation, insert multi-step adversarial perturbations

fix AMR: insert of perturbation evaluation, insert multi-step adversarial perturbations, improved the data pipeline

start experiments with command line parameters (python start_experiments.py --conf name_config_file)

released configuration files for DEMO paper (Under review at RecSys 2021)

readme file in /data/ directory with links to datasets

Source code(tar.gz)
Source code(zip)
v0.3.0(Jul 1, 2021)
ELLIOT v0.3.0 Release Notes

[v0.3.0] - 2021-06-30

Changed

early stopping strategies

offline recommendation files evaluation (ProxyRecommender, RecommendationFolder)

negative sampling evaluation

improved Microsoft Windows compatibility

binarization of explicit dataset

automatic loading of implicit datasets

multiple prefiltering

managing side information with modular loaders

alignment of side information with training data

improved Documentation: Model creation, Side Information loading, Early Stopping, Negative Sampling

added nDCG as formulated in Rendle's 2020 KDD paper

visual loader with tensorflow pipeline

added and fixing visual recsys method:

DVBPR

VBPR

DeepStyle

ACF

VNPR

added new recommender method

MF (Rendle's 2020 RecSys reproducibility paper)

EASER

RP3beta

iALS

Source code(tar.gz)
Source code(zip)
v0.2.1(Mar 27, 2021)
ELLIOT v0.2.1 Release Notes

[v0.2.1] - 2020-03-27

Changed

requirements.txt for Pillow vulnerabilities, change version to >=8.1.1

Adversarial features for ECIR tutorial "AL4Rec"

Hands-on example for ECIR Tutorial "AL4Rec"

Source code(tar.gz)
Source code(zip)
v0.2.0(Mar 12, 2021)
ELLIOT v0.2.0 Release Notes

Added

new dataloader ItemCategoryLoader

Enabled FunSVD with batch_size

Generalized feature-aware Factorization Machines

batch_size field in recommendation snippet examples

Changed

MultiDAE documentation

setup issues by @deklanw

indentation in README config by @deklanw

separated sample main and sample devel scripts

paper experiment coherence

combination Full recommendation metrics + argpartition

Dataset data mock-ups

long names namespace model generation during hyperspace exploration

default validation k setup

Source code(tar.gz)
Source code(zip)
v0.1.0(Mar 5, 2021)

Source code(tar.gz)
Source code(zip)

Elliot is a comprehensive recommendation framework that analyzes the recommendation problem from the researcher's perspective.

Related tags

Overview

ELLIOT

Installation

Install from source

CONDA

VIRTUALENV

Quick Start

Contributing

Cite

The Team

License

Acknowledgements

Comments

TensorFlow 2.4.0

Release 2.4.0

Major Features and Improvements

Breaking Changes

Release 2.3.2

Bug Fixes and Other Changes

Release 2.2.2

Bug Fixes and Other Changes

Releases(v0.3.1)

v0.3.1(Jul 8, 2021)

ELLIOT v0.3.1 Release Notes

[v0.3.1] - 2021-07-05

Changed

v0.3.0(Jul 1, 2021)

ELLIOT v0.3.0 Release Notes

[v0.3.0] - 2021-06-30

Changed

v0.2.1(Mar 27, 2021)

ELLIOT v0.2.1 Release Notes

[v0.2.1] - 2020-03-27

Changed

v0.2.0(Mar 12, 2021)

ELLIOT v0.2.0 Release Notes

Added

Changed

v0.1.0(Mar 5, 2021)

Owner

Information Systems Lab @ Polytechnic University of Bari

Global Context Enhanced Social Recommendation with Hierarchical Graph Neural Networks

Recommendation Systems for IBM Watson Studio platform

A Library for Field-aware Factorization Machines

Code for KHGT model, AAAI2021

E-Commerce recommender demo with real-time data and a graph database

Code for my ORSUM, ACM RecSys 2020, HeroGRAPH: A Heterogeneous Graph Framework for Multi-Target Cross-Domain Recommendation

Use Jupyter Notebooks to demonstrate how to build a Recommender with Apache Spark & Elasticsearch

fastFM: A Library for Factorization Machines

Spotify API Recommnder System

Deep recommender models using PyTorch.

Incorporating User Micro-behaviors and Item Knowledge 59 60 3 into Multi-task Learning for Session-based Recommendation

Code for MB-GMN, SIGIR 2021

Cross Domain Recommendation via Bi-directional Transfer Graph Collaborative Filtering Networks

A library of metrics for evaluating recommender systems

A PyTorch implementation of "Say No to the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attribute Information" (WSDM 2021)

The official implementation of "DGCN: Diversified Recommendation with Graph Convolutional Networks" (WWW '21)

Accuracy-Diversity Trade-off in Recommender Systems via Graph Convolutions

Recommender systems are the systems that are designed to recommend things to the user based on many different factors

Beyond Clicks: Modeling Multi-Relational Item Graph for Session-Based Target Behavior Prediction