A Python library for choreographing your machine learning research.

Last update: Jan 06, 2023

Overview

AI2 Tango

A Python library for choreographing your machine learning research.

⚠️ Currently in ALPHA and under active devepment ⚠️

Quick links

Installation

ai2-tango requires Python 3.7 or later.

Installing with `pip`

ai2-tango is available on PyPI. Just run

pip install ai2-tango

To install with a specific integration, such as torch for example, run

pip install ai2-tango[torch]

Installing from source

To install ai2-tango from source, first clone the repository:

git clone https://github.com/allenai/tango.git
cd tango

Then run

pip install -e .

To install with a specific integration, such as torch for example, run

pip install -e .[torch]

Team

ai2-tango is developed and maintained by the AllenNLP team, backed by the Allen Institute for Artificial Intelligence (AI2). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering. To learn more about who specifically contributed to this codebase, see our contributors page.

License

ai2-tango is licensed under Apache 2.0. A full copy of the license can be found on GitHub.

Comments

use virtualenv within Docker

Changes the Dockerfile to use a Python virtual environment so we don't mess with the system Python packages on the image. I think this is good practice in general. See https://pythonspeed.com/articles/activate-virtualenv-dockerfile/ for an explanation of how the virtual environment is activated.

opened by epwalsh 18
Make tests runnable out-of-tree for help with conda-packaging

This is the equivalent of https://github.com/allenai/allennlp/pull/5560, but for tango.

I've been carrying a simpler version of this patch since the beginning of the feedstock (since conda does not run the tests in a source checkout, but for the finished package) - this version allows introduces an override also for other users along the model introduced for allennlp.

However, as of 0.7.0 this is not complete (and I've been struggling with this in https://github.com/conda-forge/tango-feedstock/pull/8), because multicore_executor_test does not respect this infrastructure. I'm going to need some help on the latter (unless I just unconditionally skip all tests from that module) - my first impression is that test_fixtures.package.steps should be in some auxiliary module as part of tango, not in the fixtures.

opened by h-vetinari 15
initial code to serve and display tango viz

Adds initial viz for tango

notes: the allennlp team wants no build steps (no typescript, no webpack, etc) i have opted to use graphviz + d3 but not react this is admittedly a v0 and if gets any more complex, should move to react (and possibly a build process where we make an artifact in npm. for this script to use)

still to do: only showing the first run, but there are multiple runs (will be left for later)

opened by jonborchardt 14
Switches back to Steps that are responsible for their own execution and dependencies
In this PR, I went back to having the Step class keep track of its dependencies. Step graphs are represented as Step objects that reference each other.

It makes the tango-as-a-library ~~easier~~ possible.

It means that we don't have to hash all inputs to derive the name of a step. When a step is an input to another step, we only need the step name, not the step's results. This is a requirement for instantiating the step graph without executing it. In practice, it's also a requirement when step results are large and hashing them just to derive a name is not feasible.

I'm a big fan of the new way of parsing step graphs. It have moved around where that happens, but it's clearly better than trying to make FromParams do it. It also makes FromParams little simpler, though not as simple as I thought at some point.

I think (but haven't confirmed) that this approach brings back anonymous steps.

We had a problem with complex objects having some of their members initialized from steps. @epwalsh fixed it one way. I fixed it another way in here, by providing the WithUnresolvedSteps class, which defers the initialization of the complex object until the step actually runs.

I tried to leave the stuff related to the execution environment (Executor, StepGraph, StepCache, ...) alone, but wasn't able to do so completely. In particular, Executor is no longer a registrable, but that concept will have to come back soon. I just didn't want to blow up this PR. I did not keep the concept of the "step group", because I would have had to do work to keep it, and it sounds like Pete wasn't sold on that concept anymore anyways. If we need it I'm happy to bring it back properly.

I renamed the directories a bit, since we used a confusing mess of names. Now there is a "step dir", and a "work dir". The work dir is usually, but not always, inside the step dir. The work dir is scratch space. The step dir is where we store results permanently.
opened by dirkgr 13

ModuleNotFoundError when using multiprocessing in a Step

🐛 Describe the bug

I encountered ModuleNotFoundError when I try to use multiprocessing in a Step. However, when I run the same Step without using tango, everything is ok.

To replicate:

directory structure:

- test_tango
    - __init__.py
    - my_step.py

my_step.py:

from joblib import delayed, Parallel
from tango import Step

def add_one(a):
    return a+1

@Step.register("test1")
class Test1(Step):
    def run(self):
        tasks = [delayed(add_one)(i) for i in range(10)]
        results = Parallel(n_jobs=-1, prefer="processes")(tasks)
        print(results)

test_tango.jsonnet:

{
    "steps": {
        "step1": {
             "type": "test1"
        }
    }
}

Run with tango:

tango run test_tango.jsonnet -i test_tango -w workspace

got:

ERROR    [step step1] Uncaught exception                                                                                                 logging.py:373
                           joblib.externals.loky.process_executor._RemoteTraceback:
                           """
                           Traceback (most recent call last):
                             File "$PATH/lib/python3.9/site-packages/joblib/externals/loky/process_executor.py", line
                           407, in _process_worker
                               call_item = call_queue.get(block=True, timeout=timeout)
                             File "$PATH/lib/python3.9/multiprocessing/queues.py", line 122, in get
                               return _ForkingPickler.loads(res)
                           ModuleNotFoundError: No module named 'test_tango'

Directly use the Step:

>>> from test_tango.my_step import Test1
>>> runner = Test1()
>>> runner.run()
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

Everything is ok.

Versions

Python 3.9.12 ai2-tango==0.11.0

bug

opened by Tbabm 6

Error when running tango info
🐛 Describe the bug

I installed tango via pip install ai2-tango.

$ tango info Tango version 0.3.6 (python 3.9.12) Integrations: OMP: Error #15: Initializing libiomp5.dylib, but found libomp.dylib already initialized. OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/. fish: 'tango info' terminated by signal SIGABRT (Abort)

Versions

absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1654189308200/work ai2-tango @ file:///Users/runner/miniforge3/conda-bld/tango-split_1638944535621/work aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1649013201663/work aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1636093929600/work async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1640026696943/work attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work base58 @ file:///home/conda/feedstock_root/build_artifacts/base58_1635724186165/work blinker==1.4 boto3 @ file:///home/conda/feedstock_root/build_artifacts/boto3_1657783699883/work botocore @ file:///home/conda/feedstock_root/build_artifacts/botocore_1657756950790/work brotlipy @ file:///Users/runner/miniforge3/conda-bld/brotlipy_1648854211613/work cached-path==0.3.2 cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1640686991047/work certifi==2022.6.15 cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1625835329396/work charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1655906222726/work click @ file:///Users/runner/miniforge3/conda-bld/click_1651215220522/work click-help-colors @ file:///home/conda/feedstock_root/build_artifacts/click-help-colors_1638331082124/work colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1655412516417/work cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography_1657174062636/work dataclasses @ file:///home/conda/feedstock_root/build_artifacts/dataclasses_1628958434797/work datasets @ file:///home/conda/feedstock_root/build_artifacts/datasets_1635906567859/work dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1653058582944/work docker-pycreds==0.4.0 filelock==3.3.2 frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1648771747683/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1653010523205/work future @ file:///Users/runner/miniforge3/conda-bld/future_1649010190079/work gitdb @ file:///home/conda/feedstock_root/build_artifacts/gitdb_1635085722655/work GitPython @ file:///home/conda/feedstock_root/build_artifacts/gitpython_1645531658201/work google-api-core @ file:///home/conda/feedstock_root/build_artifacts/google-api-core-split_1655668757136/work google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1657692258949/work google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/work google-cloud-core @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-core_1654835241407/work google-cloud-storage @ file:///home/conda/feedstock_root/build_artifacts/google-cloud-storage_1638482791995/work google-crc32c @ file:///Users/runner/miniforge3/conda-bld/google-crc32c_1651517262819/work google-resumable-media @ file:///home/conda/feedstock_root/build_artifacts/google-resumable-media_1654213463347/work googleapis-common-protos @ file:///Users/runner/miniforge3/conda-bld/googleapis-common-protos-feedstock_1657690463235/work grpcio @ file:///Users/runner/miniforge3/conda-bld/grpcio_1645230591110/work huggingface-hub @ file:///home/conda/feedstock_root/build_artifacts/huggingface_hub_1636414419863/work idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1642433548627/work importlib-metadata @ file:///Users/runner/miniforge3/conda-bld/importlib-metadata_1653252883293/work jmespath @ file:///home/conda/feedstock_root/build_artifacts/jmespath_1655568249366/work jsonnet @ file:///Users/runner/miniforge3/conda-bld/jsonnet_1636234966467/work Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1651821407140/work mkl-fft==1.3.1 mkl-random==1.2.2 mkl-service==2.4.0 more-itertools @ file:///home/conda/feedstock_root/build_artifacts/more-itertools_1652515549581/work multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1648882472994/work multiprocess @ file:///Users/runner/miniforge3/conda-bld/multiprocess_1653595773546/work numpy @ file:///private/var/folders/sy/f16zz6x50xz3113nwtb9bvq00000gp/T/abs_e3f85f54-0572-4c0e-b190-4bc4766fc3fenewxarhq/croots/recipe/numpy_and_numpy_base_1652801682879/work oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1643507977997/work overrides @ file:///home/conda/feedstock_root/build_artifacts/overrides_1592431494961/work packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1637239678211/work pandas==1.4.1 pathtools==0.1.2 promise @ file:///Users/runner/miniforge3/conda-bld/promise_1649695782852/work protobuf==3.19.4 psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1653089238227/work pyarrow==6.0.1 pyasn1==0.4.8 pyasn1-modules==0.2.7 pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work pyDeprecate @ file:///home/conda/feedstock_root/build_artifacts/pydeprecate_1622496226936/work PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1652398519695/work pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1643496850550/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work PySocks @ file:///Users/runner/miniforge3/conda-bld/pysocks_1648857336012/work python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work pytorch-lightning @ file:///home/conda/feedstock_root/build_artifacts/pytorch-lightning_1633020752766/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1647961439546/work pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work PyYAML @ file:///Users/runner/miniforge3/conda-bld/pyyaml_1648757420486/work requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1656534056640/work requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1643557462909/work rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1637781155505/work s3transfer @ file:///home/conda/feedstock_root/build_artifacts/s3transfer_1654039987929/work sentry-sdk @ file:///home/conda/feedstock_root/build_artifacts/sentry-sdk_1657735318067/work setproctitle @ file:///Users/runner/miniforge3/conda-bld/setproctitle_1650296537432/work shortuuid @ file:///home/conda/feedstock_root/build_artifacts/shortuuid_1656529349686/work six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work smmap @ file:///home/conda/feedstock_root/build_artifacts/smmap_1611376390914/work tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1655042669710/work/tensorboard-2.9.1-py3-none-any.whl tensorboard-data-server @ file:///Users/runner/miniforge3/conda-bld/tensorboard-data-server_1649932875354/work/tensorboard_data_server-0.6.0-py3-none-macosx_10_9_x86_64.whl tensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1641458951060/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl torch==1.10.2 torchmetrics @ file:///home/conda/feedstock_root/build_artifacts/torchmetrics_1656568368092/work tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1632160078689/work typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1656706066251/work urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1657224465922/work wandb @ file:///home/conda/feedstock_root/build_artifacts/wandb_1657282472508/work Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1651670883478/work xxhash @ file:///Users/runner/miniforge3/conda-bld/python-xxhash_1649442463882/work yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1648966566248/work zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1649012893348/work
bug
opened by schmmd 6
Add Weights & Baises remote workspace
Closes #125, fixes #235, fixes #278, fixes #281

Changes proposed in this pull request:

Adds a Weights & Biases remote workspace implementation: WandbWorkspace.

Improves the WandbTrainCallback in several ways, making it work seamlessly with WandbWorkspace.

Adds Workspace.url property (must be implemented by subclasses), which returns a URL suitable to instantiate the same workspace using Workspace.from_url().

Adds Workspace.step_result_for_run() method, which gives you the result of a particular step from a particular run, given the name of the step and the name of the run (fixes one of the issues brought up in #278).

Fixes a bug with LocalWorkspace.from_parsed_url(), which resulted in the workspace using the wrong directory (fixes the other issue from #278).

Ensures deprecation warnings are displayed from the CLI (#281).

Tips for reviewers

This pull request has touched a lot of files, so here are some tips for reviewing efficiently.

Start by inspecting the first commit (3eaa483), which lays the groundwork for the subsequent commits. This commit doesn't touch anything related to Weights & Biases, but it makes some key fixes to the Step class that are needed for the WandbWorkspace, as well some fixes and additions to StepCache, Workspace, and StepInfo.

If you're satisfied with the first commit, move on to the second commit (bfd6a10), which actually adds the WandbWorkspace. I'd suggest paying close attention to the logic in WandbWorkspace and WandbStepCache, in particular.

Next, I recommend creating your own workspace on Weights & Biases, and running some mock tango experiments with the WandbWorkspace. For example:

tango run test_fixtures/experiment/hello_world.jsonnet -i test_fixtures/package -w wandb://allennlp/my-tango-testing

Browse the results on the W&B dashboard and see if you can make sense of them. If you can't, then maybe this is useless 🤨

Lastly, I recommend reviewing the documentation additions. For that, just build the docs locally and inspect the page for the wandb integration.

Before submitting

[x] I've read and followed all steps in the Making a pull request section of the CONTRIBUTING docs.

[x] I've updated or added any relevant docstrings following the syntax described in the Writing docstrings section of the CONTRIBUTING docs.

[x] If this PR fixes a bug, I've added a test that will fail without my fix.

[x] If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

[x] All GitHub Actions jobs for my pull request have passed.

Review items to complete

[x] Change naming of step artifacts to not use step name (https://github.com/allenai/tango/pull/232#discussion_r847793548)

[x] Use local timezone in timestamps, or at least when displaying times (https://github.com/allenai/tango/pull/232#discussion_r847780312)

[x] Add test with multi executor (https://github.com/allenai/tango/pull/232#discussion_r845668692)

[x] Make sure web UI works fine with a W&B workspace

[ ] Try running two workspaces in two separate threads
opened by epwalsh 6

Soft prompts

In code:

import transformers
t = transformers.AutoModel.from_pretrained("gpt2")
twp = make_prefix_transformer(t, prefix_length=3)

In config files:

{
    model: {
        type: "transformers::with_soft_prompt",
        prompt_length: 3,
        model: {
            type: "transformers::AutoModelForCausalLM::from_pretrained",
            pretrained_model_name_or_path: "gpt2"
        },
    }
}

Missing:

[x] Tests
[x] Docs
[x] Try it with T5
[x] A proper end-to-end training config that uses this
[x] Add an easy way to make only the prefix trainable, and leave the rest of the weights alone

opened by dirkgr 6

`BeakerExecutor` improvements, fix bug with `StepIndexer`
Changes proposed:

Adds field run_local to BeakerExecutor. You can use this to have the executor run certain steps locally. For instance, if you have a bunch of catwalk "tabulate-metrics" steps that are called "*-metrics" in your config, you could put run_local: ["*-metrics"] in your settings YAML file.

Adds a field gpu_type to StepResources, and the BeakerExecutor will use this to find clusters with a matching GPU type.

Fixes the unique ID for StepIndexer.
opened by epwalsh 5

Beaker Executor dies when pip temporarily loses connection

🐛 Describe the bug

https://beaker.org/ex/01GCDPHK27WET95H1CEY9DHQVV/tasks/01GCDPHK496C515GYCNVWAK3QA/job/01GCDPHPNYC4FGD6B703S0JDPV

2022-09-08T04:45:49.637609540Z Traceback (most recent call last):
2022-09-08T04:45:49.637614278Z   File "/opt/conda/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher
2022-09-08T04:45:49.637618334Z     yield
2022-09-08T04:45:49.637621543Z   File "/opt/conda/lib/python3.9/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
2022-09-08T04:45:49.637625659Z     data = self._fp.read(amt) if not fp_closed else b""
2022-09-08T04:45:49.637628946Z   File "/opt/conda/lib/python3.9/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 62, in read
2022-09-08T04:45:49.637632552Z     data = self.__fp.read(amt)
2022-09-08T04:45:49.637635690Z   File "/opt/conda/lib/python3.9/http/client.py", line 463, in read
2022-09-08T04:45:49.637639449Z     n = self.readinto(b)
2022-09-08T04:45:49.637642727Z   File "/opt/conda/lib/python3.9/http/client.py", line 507, in readinto
2022-09-08T04:45:49.637659807Z     n = self.fp.readinto(b)
2022-09-08T04:45:49.637663204Z   File "/opt/conda/lib/python3.9/socket.py", line 704, in readinto
2022-09-08T04:45:49.637666941Z     return self._sock.recv_into(b)
2022-09-08T04:45:49.637670118Z   File "/opt/conda/lib/python3.9/ssl.py", line 1241, in recv_into
2022-09-08T04:45:49.637673963Z     return self.read(nbytes, buffer)
2022-09-08T04:45:49.637677371Z   File "/opt/conda/lib/python3.9/ssl.py", line 1099, in read
2022-09-08T04:45:49.637680768Z     return self._sslobj.read(len, buffer)
2022-09-08T04:45:49.637684186Z ConnectionResetError: [Errno 104] Connection reset by peer

Does pip have retry options?

Versions

asd

bug

opened by dirkgr 5

Support Indexing for multiple return values from a Tango Step

🚀 The feature, motivation and pitch

If a Tango step returns multiple values, we currently have to pass all the returned values to another Step. The goal is to support indexing so that the user can select which returned values should be passed as an input to another Step.

Alternatives

No response

Additional context

No response
feature request

opened by isunitha98selvan 5
Update rich requirement from <13.0,>=12.3 to >=12.3,<14.0
Updates the requirements on rich to permit the latest version.

Release notes

Sourced from rich's releases.

Dropped Python3.6 support

Some relatively minor fixes and improvements. The most significant update (and the reason for the major version bump) is that Rich has dropped Python3.6 support.

If you are a Python3.6 user and can't upgrade for whatever reason, pin to version 12.6.0.

[13.0.0] - 2022-12-30

Fixed

Reversed pre and code tags in base HTML format Textualize/rich#2642

Improved detection of attrs library, that isn't confused by the presence of the attr library.

Fixed issue with locals_max_length parameter not being respected in Traceback Textualize/rich#2649

Handling of broken fileno made more robust. Fixes Textualize/rich#2645

Fixed missing fileno on FileProxy

Changed

Bumped minimum Python version to 3.7 Textualize/rich#2567

Pretty-printing of "tagged" __repr__ results is now greedy when matching tags Textualize/rich#2565

progress.track now supports deriving total from __length_hint__

Added

Add type annotation for key_separator of pretty.Node Textualize/rich#2625

Changelog

Sourced from rich's changelog.

[13.0.0] - 2022-12-30

Fixed

Reversed pre and code tags in base HTML format Textualize/rich#2642

Improved detection of attrs library, that isn't confused by the presence of the attr library.

Fixed issue with locals_max_length parameter not being respected in Traceback Textualize/rich#2649

Handling of broken fileno made more robust. Fixes Textualize/rich#2645

Fixed missing fileno on FileProxy

Changed

Bumped minimum Python version to 3.7 Textualize/rich#2567

Pretty-printing of "tagged" __repr__ results is now greedy when matching tags Textualize/rich#2565

progress.track now supports deriving total from __length_hint__

Added

Add type annotation for key_separator of pretty.Node Textualize/rich#2625

[12.6.0] - 2022-10-02

Added

Parse ANSI escape sequences in pretty repr Textualize/rich#2470

Add support for FORCE_COLOR env var Textualize/rich#2449

Allow a max_depth argument to be passed to the install() hook Textualize/rich#2486

Document using None as name in __rich_repr__ for tuple positional args Textualize/rich#2379

Add font_aspect_ratio parameter in SVG export Textualize/rich#2539

Added Table.add_section method. Textualize/rich#2544

Fixed

Handle stdout/stderr being null Textualize/rich#2513

Fix NO_COLOR support on legacy Windows Textualize/rich#2458

Fix pretty printer handling of cyclic references Textualize/rich#2524

Fix missing mode property on file wrapper breaking uploads via requests Textualize/rich#2495

Fix mismatching default value of parameter ensure_ascii Textualize/rich#2538

Remove unused height parameter in Layout class Textualize/rich#2540

Fixed exception in Syntax.rich_measure for empty files

Changed

Removed border from code blocks in Markdown

[12.5.2] - 2022-07-18

Added

... (truncated)

Commits

d14b304 Merge pull request #2727 from Textualize/v13.0.0

7cf0797 changelog

245c737 release in changelog

b2fa5f1 changelog

7601290 Merge pull request #2715 from Textualize/win-batch

e833da9 don't reraise

b66f1f0 comment

c8c116c fix typing

1f3f18c try new poetry

497177f better batching

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 0
Bump sphinx from 5.3.0 to 6.0.0
Bumps sphinx from 5.3.0 to 6.0.0.

Release notes

Sourced from sphinx's releases.

v6.0.0

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b2

Changelog: https://www.sphinx-doc.org/en/master/changes.html

v6.0.0b1

Changelog: https://www.sphinx-doc.org/en/master/changes.html

Changelog

Sourced from sphinx's changelog.

Release 6.0.0 (released Dec 29, 2022)

Dependencies

#10468: Drop Python 3.6 support

#10470: Drop Python 3.7, Docutils 0.14, Docutils 0.15, Docutils 0.16, and Docutils 0.17 support. Patch by Adam Turner

Incompatible changes

#7405: Removed the jQuery and underscore.js JavaScript frameworks.

These frameworks are no longer be automatically injected into themes from Sphinx 6.0. If you develop a theme or extension that uses the jQuery, $, or $u global objects, you need to update your JavaScript to modern standards, or use the mitigation below.

The first option is to use the sphinxcontrib.jquery_ extension, which has been developed by the Sphinx team and contributors. To use this, add sphinxcontrib.jquery to the extensions list in conf.py, or call app.setup_extension("sphinxcontrib.jquery") if you develop a Sphinx theme or extension.

The second option is to manually ensure that the frameworks are present. To re-add jQuery and underscore.js, you will need to copy jquery.js and underscore.js from the Sphinx repository_ to your static directory, and add the following to your layout.html:

.. code-block:: html+jinja

{%- block scripts %} {{ super() }} {%- endblock %}

.. _sphinxcontrib.jquery: https://github.com/sphinx-contrib/jquery/

Patch by Adam Turner.

#10471, #10565: Removed deprecated APIs scheduled for removal in Sphinx 6.0. See :ref:dev-deprecated-apis for details. Patch by Adam Turner.

#10901: C Domain: Remove support for parsing pre-v3 style type directives and roles. Also remove associated configuration variables c_allow_pre_v3 and c_warn_on_allowed_pre_v3. Patch by Adam Turner.

Features added

... (truncated)

Commits

5b56a23 Bump to 6.0.0 final

f1d1e9c Update coverage workflow for Tox 4

66a738c Update coverage workflow for new configuration location

041e5f8 Add test coverage for 'today_fmt' reference substitution (#10980)

da25145 Remove unnecessary conditional import in sphinx.ext.napoleon (#11043)

45a0ea9 Migrate coveragepy config into pyproject.toml (#11025)

3ec54f1 Create a pydata_sphinx_theme section in usage examples (#11046)

32bce8f Copy edit the tutorial (#11049)

9844162 Fix example using add_config_value (#10937)

bf4a626 RTD builder: add graphviz depedendency (#11040)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 0
Bump isort from 5.10.1 to 5.11.4
Bumps isort from 5.10.1 to 5.11.4.

Release notes

Sourced from isort's releases.

5.11.4

Changes

Remove obsolete toml import from the test suite (#1978) @mgorny

Stop installing documentation files to top-level site-packages (#2057) @mgorny

Only run release workflows for upstream (#2052) @hugovk

:package: Dependencies

Bump Poetry 1.3.1 (#2058) @staticdev

5.11.3

Changes

Renable portray (#2043) @timothycrosley

chore(ci): add minimum GitHub token permissions for workflows (#1969) @varunsh-coder

:beetle: Fixes

Fix packaging pypoetry (#2042) @staticdev

Fix settings for py3.11 (#2040) @staticdev

:construction_worker: Continuous Integration

General CI improvements (#2041) @staticdev

Add release workflow (#2026) @staticdev

v5.11.3

Changes

Renable portray (#2043) @timothycrosley

chore(ci): add minimum GitHub token permissions for workflows (#1969) @varunsh-coder

:beetle: Fixes

Fix packaging pypoetry (#2042) @staticdev

Fix settings for py3.11 (#2040) @staticdev

:construction_worker: Continuous Integration

General CI improvements (#2041) @staticdev

Add release workflow (#2026) @staticdev

5.11.2

Changes

Hotfix for --version. (#2035) @felixxm

5.11.1

Changes December 12 2022

... (truncated)

Changelog

Sourced from isort's changelog.

5.11.4 December 21 2022

Fixed #2038 (again): stop installing documentation files to top-level site-packages (#2057) @mgorny

CI: only run release workflows for upstream (#2052) @hugovk

Tests: remove obsolete toml import from the test suite (#1978) @mgorny

CI: bump Poetry 1.3.1 (#2058) @staticdev

5.11.3 December 16 2022

Fixed #2007: settings for py3.11 (#2040) @staticdev

Fixed #2038: packaging pypoetry (#2042) @staticdev

Docs: renable portray (#2043) @timothycrosley

Ci: add minimum GitHub token permissions for workflows (#1969) @varunsh-coder

Ci: general CI improvements (#2041) @staticdev

Ci: add release workflow (#2026) @staticdev

5.11.2 December 12 2022

Hotfix #2034: isort --version is not accurate on 5.11.x releases (#2034) @gschaffner

5.11.1 December 12 2022

Hotfix #2031: only call colorama.init if colorama is available (#2032) @tomaarsen

5.11.0 December 12 2022

Added official support for Python 3.11 (#1996, #2008, #2011) @staticdev

Dropped support for Python 3.6 (#2019) @barrelful

Fixed problematic tests (#2021, #2022) @staticdev

Fixed #1960: Rich compatibility (#1961) @ofek

Fixed #1945, #1986: Python 4.0 upper bound dependency resolving issues @staticdev

Fixed Pyodide CDN URL (#1991) @andersk

Docs: clarify description of use_parentheses (#1941) @mgedmin

Fixed #1976: black compatibility for .pyi files @XuehaiPan

Implemented #1683: magic trailing comma option (#1876) @legau

Add missing space in unrecoverable exception message (#1933) @andersk

Fixed #1895: skip-gitignore: use allow list, not deny list @bmalehorn

Fixed #1917: infinite loop for unmatched parenthesis (#1919) @anirudnits

Docs: shared profiles (#1896) @matthewhughes934

Fixed build-backend values in the example plugins (#1892) @mgorny

Remove reference to jamescurtin/isort-action (#1885) @AndrewLane

Split long cython import lines (#1931) @davidcollins001

Update plone profile: copy of black, plus three settings. (#1926) @mauritsvanrees

Fixed #1815, #1862: Add a command-line flag to sort all re-exports (#1863) @parafoxia

Fixed #1854: lines_before_imports appending lines after comments (#1861) @legau

Remove redundant multi_line_output = 3 from "Compatibility with black" (#1858) @jdufresne

Add tox config example (#1856) @umonaca

Docs: add examples for frozenset and tuple settings (#1822) @sgaist

Docs: add multiple config documentation (#1850) @anirudnits

Commits

98390f5 Merge pull request #2059 from PyCQA/version/5.11.4

df69a05 Bump version 5.11.4

f9add58 Merge pull request #2058 from PyCQA/deps/poetry-1.3.1

36caa91 Bump Poetry 1.3.1

3c2e2d0 Merge pull request #1978 from mgorny/toml-test

45d6abd Remove obsolete toml import from the test suite

3020e0b Merge pull request #2057 from mgorny/poetry-install

a6fdbfd Stop installing documentation files to top-level site-packages

ff306f8 Fix tag template to match old standard

227c4ae Merge pull request #2052 from hugovk/main

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 0
Add ability to display the executor output
Fixes #487

Changes proposed in this pull request: Lightly modified version of code pointed by @epwalsh here

Adds the ability to display the executor output. Can be used as follows:

output = executor.execute_step_graph(step_graph) output.display()

Tested locally and produces an output as follows:

Before submitting

[x] I've read and followed all steps in the Making a pull request section of the CONTRIBUTING docs.

[ ] I've updated or added any relevant docstrings following the syntax described in the Writing docstrings section of the CONTRIBUTING docs.

[ ] If this PR fixes a bug, I've added a test that will fail without my fix.

[ ] If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

[ ] All GitHub Actions jobs for my pull request have passed.
opened by BigRedT 1
Bump black from 22.10.0 to 22.12.0
Bumps black from 22.10.0 to 22.12.0.

Release notes

Sourced from black's releases.

22.12.0

Preview style

Enforce empty lines before classes and functions with sticky leading comments (#3302)

Reformat empty and whitespace-only files as either an empty file (if no newline is present) or as a single newline character (if a newline is present) (#3348)

Implicitly concatenated strings used as function args are now wrapped inside parentheses (#3307)

Correctly handle trailing commas that are inside a line's leading non-nested parens (#3370)

Configuration

Fix incorrectly applied .gitignore rules by considering the .gitignore location and the relative path to the target file (#3338)

Fix incorrectly ignoring .gitignore presence when more than one source directory is specified (#3336)

Parser

Parsing support has been added for walruses inside generator expression that are passed as function args (for example, any(match := my_re.match(text) for text in texts)) (#3327).

Integrations

Vim plugin: Optionally allow using the system installation of Black via let g:black_use_virtualenv = 0(#3309)

Changelog

Sourced from black's changelog.

22.12.0

Preview style

Enforce empty lines before classes and functions with sticky leading comments (#3302)

Reformat empty and whitespace-only files as either an empty file (if no newline is present) or as a single newline character (if a newline is present) (#3348)

Implicitly concatenated strings used as function args are now wrapped inside parentheses (#3307)

Correctly handle trailing commas that are inside a line's leading non-nested parens (#3370)

Configuration

Fix incorrectly applied .gitignore rules by considering the .gitignore location and the relative path to the target file (#3338)

Fix incorrectly ignoring .gitignore presence when more than one source directory is specified (#3336)

Parser

Parsing support has been added for walruses inside generator expression that are passed as function args (for example, any(match := my_re.match(text) for text in texts)) (#3327).

Integrations

Vim plugin: Optionally allow using the system installation of Black via let g:black_use_virtualenv = 0(#3309)

Commits

2ddea29 Prepare release 22.12.0 (#3413)

5b1443a release: skip bad macos wheels for now (#3411)

9ace064 Bump peter-evans/find-comment from 2.0.1 to 2.1.0 (#3404)

19c5fe4 Fix CI with latest flake8-bugbear (#3412)

d4a8564 Bump sphinx-copybutton from 0.5.0 to 0.5.1 in /docs (#3390)

2793249 Wordsmith current_style.md (#3383)

d97b789 Remove whitespaces of whitespace-only files (#3348)

c23a5c1 Clarify that Black runs with --safe by default (#3378)

8091b25 Correctly handle trailing commas that are inside a line's leading non-nested ...

ffaaf48 Compare each .gitignore found with an appropiate relative path (#3338)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies python
opened by dependabot[bot] 0

Releases(v1.1.0)

v1.1.0(Dec 1, 2022)
What's new

Added 🎉

Added gpu_type field to StepResources. The BeakerExecutor can use this to determine which clusters to a submit a step to.

Added machine field to StepResources. You can set this to "local" when using the BeakerExecutor to force it to run the step locally.

Added --ext-var argument to tango run for setting JSONNET external variables when loading the experiment config.

Added @step() decorator to create Step classes from functions.

Added the transformers::with_soft_prompt integration, to make soft-prompted prefix transformers easy.

Removed 👋

Removed PyTorch Lightning integration.

Removed tango server command and --serve/--no-serve option for tango run.

Removed source_release.py, which was checked in by accident.

Fixed ✅

Fixed issue where Executor parallelism option in a Tango settings file would be ignored.

Fixed a bug where the unique ID of a step that depends on a key-value of the result of another step could change if the name of the other step changes.

Fixed a bug where importing certain libraries (like torchmetrics) would mess with our exception handling because they set sys.excepthook for some reason. Now we always reset sys.excepthook after importing.

The type hints for the flax trainer suggested that the training split is optional when in fact it's mandatory.

Made BeakerWorkspace / BeakerStepLock more robust when a job is preempted.

Minor performance improvements for the Beaker executor and workspace.

Commits

73bfa86 Soft prompts (#231) 79b7d01 Beaker integration perf improvements d455541 Train is not optional (#474) 3eab580 Beaker integration perf improvements (#475) 241b4eb Remove step that was never supposed to be there (#478) 5f5ba41 Add @step() decorator (#476) c0c4ae0 Make BeakerStepLock robust to preempted jobs 39d3d66 Remove tango server (#470) fef9bba reset sys.excepthook after importing other modules, add --ext-var CLI option (#471) ccebb4c Automatically remove ephemeral Beaker datasets 81d773e BeakerExecutor improvements, fix bug with StepIndexer (#469) c05a80a Bump sphinx-copybutton from 0.5.0 to 0.5.1 (#467) 147e408 Remove PyTorch Lightning integration (#468) cd4f626 Update more-itertools requirement from <9.0,>=8.0 to >=8.0,<10.0 (#456) d219dff Fix doc build failure ee71c09 Fix bug with setting parallelism in settings file (#466) 1609eb4 Bump mypy from 0.982 to 0.991 (#465)
Source code(tar.gz)
Source code(zip)
ai2-tango-1.1.0.tar.gz(178.16 KB)
ai2_tango-1.1.0-py3-none-any.whl(219.32 KB)
v1.0.2(Nov 15, 2022)
What's new

Changed ⚠️

BeakerScheduler can now return a list of clusters.

Commits

64215d8 Bump mypy from 0.982 to 0.990 (#463) acc1082 Update torch requirement from <1.13,>=1.9 to >=1.9,<1.14 (#460) 570d24e Use new constraints field for cluster assignment (#462) 5afd89f Set Beaker client User-Agent header to Tango v*.. (#459) 9d9628f Fix progress logging statement from BeakerExecutor (#458)
Source code(tar.gz)
Source code(zip)
ai2-tango-1.0.2.tar.gz(189.10 KB)
ai2_tango-1.0.2-py3-none-any.whl(235.47 KB)
v1.0.1(Oct 20, 2022)
What's new

Fixed ✅

LightningTrainStep now can take a Lazy model object which results in a gauranteed deterministic hash.

Fixed issue where remote Workspace implementations like WandbWorkspace and BeakerWorkspace would use the same local cache regardless of the W&B / Beaker workspace being used.

Fixed bug with TorchEvalStep when constructing callbacks.

Fixed some import error issues caused when an integration is not installed.

Fix incorrect reporting of final results in MulticoreExecutor.

Changed ⚠️

Wandb step cache retries api call in case of timeout

beaker-py >= 1.11 required.

Commits

26e6416 Bump sphinx from 5.2.3 to 5.3.0 (#455) c10cec5 Retry wandb call (#450) f950b12 Use separate local cache dirs for each workspace (#451) 4c70161 Uncacheable step failure should be reported (#453) 9476942 Bump black from 22.8.0 to 22.10.0 (#445) 821c0fc Bump furo from 2022.9.15 to 2022.9.29 (#435) 8d0b146 Allow Lazy model input to LightningTrainStep (#448) 12eec56 Fix import issues when missing integrations (#447) d16997c Fix bug with TorchEvalStep (#442) 9a5f792 Bump beaker-py to version >=1.11 (#443) f5c2e52 Add to FAQ (#440)
Source code(tar.gz)
Source code(zip)
ai2-tango-1.0.1.tar.gz(189.03 KB)
ai2_tango-1.0.1-py3-none-any.whl(235.51 KB)
v1.0.0(Oct 5, 2022)
This is the first stable release of AI2 Tango, the culmination of over a year's work and nearly 1000 commits!

We've been working on this project quietly for a while on the AllenNLP team, and it's being used daily by researchers here. So we're excited to officially announce it now that we're happy with the API 🎉

What's new since the last release

Added 🎉

Added step_extra_dependencies input field to Step class that can be used to force a dependency on another step even if the current step doesn't directly depend on the output of the other step. See #418 for more context.

Changed ⚠️

beaker-py >= 1.10 required.

Fixed ✅

Long log lines will be soft-wrapped to ensure that links are clickable.

Fixed a bug where some workspaces could be left in a bad state if a step's Format failed to serialize the step's result in Workspace.step_finished().

Sometimes functions and methods end up as arguments to steps, which means we have to hash them. Instead of taking a hash of the function, we now take a hash of the function's module and name.

Fixed a bug with the Beaker executor where it would hang at the end of a run if a step failed that is a dependency of another step.

Fixed tests to work with new version of transformers.

Fixed Executor.execute_sub_graph_for_step() to be able to run the step's dependencies in parallel.

Commits

a48a825 Only install gh in entrypoint if needed (#439) 69b948d Improve error handling for step state edge case (#429) 39b3132 Fix Executor.execute_sub_graph_for_step() (#438) 408944e Loosen range on some dependencies (#437) 76cd46d Follow up fix for #401 (#434) 04e7963 Fix bug with Beaker executor (#430) fb077e6 Bump fairscale from 0.4.9 to 0.4.11 (#432) 25f7373 Bump furo from 2022.6.21 to 2022.9.15 (#410) f11925c Bump mypy from 0.971 to 0.982 (#433) 4845317 Bump sphinx from 5.1.1 to 5.2.3 (#431) 572f83b Bump myst-parser from 0.18.0 to 0.18.1 (#426) e1fc2d3 Add step_extra_dependencies option to Step class (#419) 19927d3 Don't pin protobuf anymore (#428) 3dbe8c7 We need click 8 to work. (#427) cbdbe68 Hashing functions (#424) 747469f Mark Format serialization failures as step failure (#421) b1d6431 Ensure base path for included modules kept in sys.path (#406) e1a1cd1 Minor improvements to BeakerExecutor internals (#416) d3e1891 Ensure log lines get soft-wrapped so links are always clickable (#415)
Source code(tar.gz)
Source code(zip)
ai2-tango-1.0.0.tar.gz(188.03 KB)
ai2_tango-1.0.0-py3-none-any.whl(234.52 KB)
v0.14.0(Sep 20, 2022)
What's new

Added 🎉

Adds a function to modify a Hugging Face transformer with IA3 adaptors

Added a BeakerScheduler registrable class, specified as the argument scheduler to BeakerExecutor, which controls the resources assigned to steps ran on Beaker. Users can implement their own BeakerScheduler subclasses to customize the resource assignment behavior.

Changed ⚠️

In the tango run command, --no-server is now the default. Use --server to start the server.

Fixed ✅

Made BeakerExecutor more robust to connection, timeout, SSL, and other recoverable HTTP errors.

Made the BeakerStepLock more robust, and as a result BeakerWorkspace is more robust and should require less manual intervention for locks in a bad state.

Fixed a bug with the internal scheduling logic of the BeakerExecutor which could delay submitting some steps in parallel.

Fixed a bug where creating a StepInfo object from params might result in unnecessary imports.

Fixed a bug where canceling the Beaker executor might not work properly.

Fixed a bug where the trainer trains too much when train_epochs is set and you're using gradient accumulation.

Fixed how the results of uncacheable steps are displayed by tango run.

Beaker executor won't run duplicate cacheable steps at the same time.

Commits

0828adc BeakerExecutor won't run duplicate cacheable steps (#414) 7382019 IA3 adaptors (#403) d498cf7 Hot fix to final output bff9ebf Add warning when steps can't be run yet, bug fixes (#408) c72552e Don't start server by default (#409) d34fe09 Added BeakerScheduler class for handling resource assignment (#407) 5dcbb56 Gradient accumulation and train_epochs (#402) d27bbef Make BeakerStepLock more robust (#401) cd9b5fd Fix bug with StepInfo.from_params(), canceling BeakerExecutor, reserve "ref" name (#400) 6ff6b9e Fix bug with scheduling logic (#399) 15196f2 Deterministic hashing for tensors (#398) 230d78e Make BeakerExecutor more robust to all recoverable errors types (connection, HTTP, SSL, timeout, etc) (#397) 5f63a27 Bump fairscale from 0.4.8 to 0.4.9 (#391)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.14.0.tar.gz(187.13 KB)
ai2_tango-0.14.0-py3-none-any.whl(233.66 KB)
v0.13.0(Sep 8, 2022)
What's new

Added 🎉

You can now reference into a particular index of the result of another step in a config. For example: {type: "ref", ref: "some_previous_step", key: 0}. The key field can be an integer if the result of the referenced step is a list or tuple, or a string if the result of the referenced step is a dictionary.

Added priority parameter to Beaker executor for setting the default task priority for Beaker jobs.

Added Workspace.step_result() method for getting a step's result from the latest run.

tango run will now display a URL to the logs for failed steps when you use the BeakerExecutor.

Changed ⚠️

The TorchTrainStep now enables monitoring arbitrary model outputs during training. TorchTrainEngine.forward_train now returns a tuple loss, model_outputs for each micro batch and the list of model outputs for all micro batches in a batch is passed to the TrainCallback.log_batch and TrainCallback.post_batch.

Tango will now automatically search Python modules in the current working directory for registered classes so that you don't always need to use the --include-package setting.

The minimum supported Python version is now 3.8.

Added support for PyTorch Lightning 1.7.x

The Beaker Executor will no-longer live-stream logs from Beaker jobs, but logs will be viewable on Beaker and more readable.

Only the Beaker executor requires a clean working directory

Fixed ✅

Fixed a bug that did not allow a wandb artifact's type to be set from a step's metadata dictionary.

Fixed a bug with how the Beaker executor streams log lines from Beaker which sometimes resulted in messages missing some starting characters, and tqdm lines being duplicated.

Fixed a bug in the Beaker workspace where the lock dataset wouldn't be removed if the step was found to be in an invalid state.

Improved cluster choice logic in BeakerExecutor to ensure greater diversity of clusters when submitting many steps at once.

Fixed bug where sub-processes of the multicore executor would use the wrong executor if executor was defined in a tango.yml file.

Commits

4f89d55 Improve Beaker cluster choice logic (#392) e1ceae2 Display URL to logs for failed steps (#390) 3dc9591 Bump black from 22.6.0 to 22.8.0 (#380) c9ce257 Catch when Beaker experiments are stopped (#389) 0fe12e9 Fix issues with WandbWorkspace causing CI crash (#388) 342eb26 Keep parameters in Params objects to make error messages more readable (#375) 92f0354 Simplified beaker logging (#383) fd9d3cc Only the Beaker executor needs clean working directories (#373) 06f26ae Update wandb artifact type (#378) f6a6b70 Update base images, get us out of the latest infinite loop of pip madness (#382) 306986b Catch all errors when attempting log record decode (#379) 628caff Allowing indexing into step results in config (#371) 858cef8 Minor improvement to Beaker logging (#377) 7a5619e Add Workspace.step_result() method (#374) 0750d76 Fix bugs with how BeakerExecutor streams logs (#372) 6e8b107 Detailed train outputs (#369) bcd50d8 Update pytorch-lightning requirement from <1.7,>=1.6 to >=1.6,<1.8 (#349) 8ed0c86 Bump fairscale from 0.4.6 to 0.4.8 (#347) 62f2746 Python minimum version is 3.8 (#368) 45e02fe Auto import local Python modules when searching for registered classes (#367)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.13.0.tar.gz(181.61 KB)
ai2_tango-0.13.0-py3-none-any.whl(227.95 KB)
v0.12.0(Aug 24, 2022)
What's new

Added 🎉

Step resources:

Added a step_resources parameter to the Step class which should be used to describe the computational resources required to run a step. Executor implementations can use this information. For example, if your step needs 2 GPUs, you should set step_resources=StepResources(gpu_count=2) ("step_resources": {"gpu_count": 2} in the configuration language).

Added a Step.resources() property method. By default this returns the value specified by the step_resources parameter. If your step implementation always requires the same resources, you can just override this method so you don't have to provide the step_resources parameter.

Step execution:

Added an executor field to the tango.yml settings. You can use this to define the executor you want to use by default.

Added a Beaker Executor to the Beaker integration, registered as an Executor with the name "beaker". To use this executor, add these lines to your tango.yml file:
executor: type: beaker beaker_workspace: ai2/my-workspace clusters: - ai2/general-cirrascale

See the docs for the BeakerExecutor for more information on the input parameters.

Step class:

Added a metadata field to the step class API. This can be set through the class variable METADATA or through the constructor argument step_metadata.

Weights & Biases integration:

You can now change the artifact kind for step result artifacts by adding a field called "artifact_kind" to a step's metadata. For models, setting "artifact_kind" to "model" will add the corresponding artifact to W&B's new model zoo.

Changed ⚠️

CLI:

The tango run command will throw an error if you have uncommitted changes in your repository, unless you use the --allow-dirty flag.

The tango run command will use the lightweight base executor (single process) by default. To use the multi-process executor, set -j/--parallelism to 1 or higher or -1 to use all available CPU cores.

Fixed ✅

Fixed bug where StepInfo environment and platform metadata could be out-of-date if a step is run again due to failure.

Fixed a bug where an unfortunate combination of early stopping and decreasing model performance could result in a crash in the torch trainer.

Commits

befb00a Add workspace_metadata arg to Step class, allow changing artifact kind in W&B workspace (#363) 5ab1c2a Fix undefined behavior with TorchTrainStep (#366) bf3c1a0 Update filelock requirement from <3.8,>=3.4 to >=3.4,<3.9 (#354) b4e48a7 Update jsonpickle requirement from <2.2.0,>=2.1.0 to >=2.1.0,<2.3.0 (#351) 1c491f0 Update wandb requirement from <0.13,>=0.12 to >=0.12,<0.14 (#350) 93d5eb4 Bump allenai/setup-beaker from 1 to 2 (#359) dc0f89a Fix #355 - ensure git metadata is up-to-date (#361) 258e880 Raise better error msg from step_result_for_run() (#360) 43916d1 Print debugging information about the repo used. (#353) 928aa7a Add BeakerExecutor (#340)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.12.0.tar.gz(174.62 KB)
ai2_tango-0.12.0-py3-none-any.whl(226.74 KB)
v0.11.0(Aug 4, 2022)
What's new

Added 🎉

Added a Flax integration along with an example config.

Commits

b4cd2b3 Flax Integration (#313) b9a7422 Bump sphinx from 5.0.2 to 5.1.1 (#346) d7952ef Bump mypy from 0.961 to 0.971 (#339) 6a58bfd Put PIP install instructions first (#348)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.11.0.tar.gz(163.01 KB)
ai2_tango-0.11.0-py3-none-any.whl(214.34 KB)
v0.10.1(Jul 26, 2022)
What's new

Fixed ✅

Fixed issue where the StepInfo config argument could be parsed into a Step.

Restored capability to run tests out-of-tree.

Commits

2498318 Fix issue where StepInfo config could be parsed into a Step (#344) 57096b2 Make tests runnable out-of-tree for help with conda-packaging (#307)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.10.1.tar.gz(151.15 KB)
ai2_tango-0.10.1-py3-none-any.whl(194.88 KB)
v0.10.0(Jul 7, 2022)
What's new

Changed ⚠️

Renamed workspace parameter of BeakerWorkspace class to beaker_workspace.

Executor class is now a Registrable base class. MulticoreExecutor is registered as "multicore".

Removed 👋

Removed StepExecutionMetadata. Its fields have been absorbed into StepInfo.

Fixed ✅

Improved Step.ensure_result() such that the step's result doesn't have to be read from the cache.

Fixed an issue with the output from MulticoreExecutor such that it's now consistent with the default Executor for steps that were found in the cache.

One of our error messages referred to a configuration file that no longer exists.

Improved performance of BeakerWorkspace.

Added 🎉

Added the ability to train straight Model instead of just Lazy[Model]

Commits

4e809f5 Eager models (#319) 361777b Metadata changes, make executor registrable (#331) a6b0be9 Beaker workspace performance (#328) f43e5ea Update torch requirement from <1.12,>=1.9 to >=1.9,<1.13 (#330) 8495c64 update dev dependencies (#333) 712d862 Make multicore executor output consistent with default (#325) 903569c Refer to the right config file (#324) bd9e4be Modernize our issue templates (#323)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.10.0.tar.gz(149.94 KB)
ai2_tango-0.10.0-py3-none-any.whl(193.25 KB)
v0.9.1(Jun 24, 2022)
What's new

Fixed ✅

Fixed non-deterministic behavior in TorchTrainStep.

Fixed bug in BeakerWorkspace where .step_info(step) would raise a KeyError if the step hasn't been registered as part of a run yet.

Fixed a bug in BeakerWorkspace where it would send too many requests to the beaker service.

Fixed a bug where WandbWorkspace.step_finished() or .step_failed() would crash if called from a different process than .step_starting().

Fixed a bug in WandbWorkspace.step_finished() which led to a RuntimeError sometimes while caching the result of a step.

Commits

c6fc5be Fix bugs with Workspace and WandbWorkspace, specifically (#321) 80c90ca Beaker DOS fix (#315) 8b75591 Log from BeakerStepLock at WARNING level (#316) 4d46d67 fix non-deterministic behavior in TorchTrainStep (#314) c59b6b3 Bump actions/setup-python from 3 to 4 (#311) b02cf40 Bump sphinx from 4.5.0 to 5.0.1 (#305) 4501815 Bump furo from 2022.6.4 to 2022.6.4.1 (#309) da9c29c Fix bug in Beaker workspace (#312) e8422cb Bump mypy from 0.960 to 0.961 (#308) 8256a74 Bump myst-parser from 0.17.2 to 0.18.0 (#310) 44ae92e Bump furo from 2022.4.7 to 2022.6.4 (#306) 39923ae Update protobuf requirement from <=3.20.0 to <4.22.0 (#301) e7ef1f5 Registerables first steps eg (#304)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.9.1.tar.gz(149.61 KB)
ai2_tango-0.9.1-py3-none-any.whl(192.82 KB)
v0.9.0(Jun 1, 2022)
What's new

Added 🎉

Added a Beaker integration that comes with BeakerWorkspace, a remote Workspace implementation that uses Beaker Datasets under the hood.

Added a datasets::dataset_remix step that provides the split remixing functionality of tango.steps.datasest_remix.DatasetRemixStep now for Huggingface DatasetDict.

Changed ⚠️

If you try to import something from a tango integration that is not fully installed due to missing dependencies, an IntegrationMissingError will be raised instead of ModuleNotFound.

You can now set -j 0 in tango run to disable multicore execution altogether.

Fixed ✅

Improved how steps and workspaces handle race conditions when different processes are competing to execute the same step. This would result in a RuntimeError before with most workspaces, but now it's handled gracefully.

Fixed bug which caused GradScaler state to not be saved and loaded with checkpoints.

Commits

0ddd2ac Add Beaker integration (#296) 6bdd1dd Updates the Euler example (#297) bc89470 GradScaler state saving and loading (#293) b8562db fix old filename in CONTRIBUTING.md (#300) 4aff1bb Dataset remix (#298) eb1fcd8 Bump mypy from 0.950 to 0.960 (#295) 903741e Update filelock requirement from <3.7,>=3.4 to >=3.4,<3.8 (#284) b58b823 Handle missing integrations (#292)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.9.0.tar.gz(149.61 KB)
ai2_tango-0.9.0-py3-none-any.whl(192.74 KB)
v0.8.0(May 20, 2022)
What's new

Added 🎉

Added a Weights & Baises remote Workspace implementation: WandbWorkspace, registered as "wandb". This can be instantiated from a workspace URL in the form "wandb://entity/project".

Added a method Workspace.step_result_for_run which gives the result of a step given the run name and step name within that run.

Added property Workspace.url, which returns a URL for the workspace that can be used to instantiate the exact same workspace using Workspace.from_url(). Subclasses must implement this.

Changed ⚠️

StepInfo start and end times will be always be in UTC now.

WandbTrainCallback now logs system metrics from each worker process in distributed training.

StepCache.__contains__() and StepCache.__getitem__() now take accept either a Step or StepInfo as an argument (Union[Step, StepInfo]).

Refactored tango.step_graph.StepGraph to allow initialization from a Dict[str, Step].

Executor.execute_step_graph() now attempts to execute all steps and summarizes success/failures.

Fixed ✅

Fixed bug with LocalWorkspace.from_parsed_url() (#278).

Deprecation warnings will now be logged from tango CLI.

Fixed the text format in the case of serializing an iterator of string.

Added missing default value of None to TangoGlobalSettings.find_or_default().

Mypy has become incompatible with transformers and datasets, so we have to disable the checks in some places.

The VERSION member of step arguments that were wrapped in Lazy were not respected. Now they are.

Commits

3069226 Makes sure the VERSION parameter of classes is respected even when we construct them inside of a Lazy object. (#289) dd71446 Add Weights & Baises remote workspace (#232) e3f2bd2 Adds a dependency that's missing from transformers (#285) 25919e1 Fixes the text format (#283) 381de74 Add missing default to TangoGlobalSettings.find_or_default() (#282) 9ac708a Update click requirement from <8.1.3,>=7.0 to >=7.0,<8.1.4 (#277) 749357e Bump mypy from 0.942 to 0.950 (#276) 2c59c96 Bump allenai/beaker-run-action from 1.0 to 1.1 (#274) 53ffe80 refactor (#275)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.8.0.tar.gz(142.29 KB)
ai2_tango-0.8.0-py3-none-any.whl(183.19 KB)
v0.7.0(Apr 19, 2022)
What's new

Added 🎉

Added the "-n/--name" option to tango run. This option allows the user to give the run an arbitrary name.

Added a convenience property .workspace to Step class that can be called from a step's .run() method to get the current Workspace being used.

Gave FromParams objects (which includes all Registrable objects) the ability to version themselves.

Added CLI option to run a single step in a config using --step-name or -s.

Added a MultiCoreExecutor that executes steps in parallel.

Added an ExecutorOutput dataclass that is returned by Executor.execute_step_graph().

StepGraph now prints itself in a readable way.

Tango now automatically detects when it's running under a debugger, and disables multicore support accordingly. Many debuggers can't properly follow sub-processes, so this is a convenience for people who love debuggers.

Added more models to the stuff we can import from the transformers library.

Added new example for finetuning text-to-text models.

Changed ⚠️

Renamed click_logger to cli_logger, and we now use rich's logging Handler as the default handler, which means prettier output, better tracebacks, and you can use rich's markup syntax with the cli_logger to easily add style to text.

Refactored tango.step_graph.StepGraph to allow initialization from a Dict[str, Step].

Executor.execute_step_graph() now attempts to execute all steps and summarizes success/failures.

Upgraded PyTorch version in tango Docker image to latest v1.11.0+cu113.

RunGeneration now allows model object as input.

Fixed ✅

Fixed bug that mistakenly disallowed fully-qualified names containing "_" (underscores) in the config.

Fixed bug where TorchTrainStep working directory would be left in an unrecoverable state if training failed after saving the final model weights.

Fixed bug in FromParams where **kwargs might be passed down to the constructors of arguments.

Fixed bug in the way dependencies are tracked between steps.

Fixed bug that caused MulticoreExecutor to hang in case of a failing step that was required recursively (not directly) downstream.

Fixed bug in the way dependencies are tracked between steps

Compatibility with PyTorch Lightning 1.6

Commits

1083049 Finetuning (#255) 42b1dba Bug fix with failing steps (#257) 7bd251a Bump myst-parser from 0.17.0 to 0.17.2 (#273) cc9a1dd Bump actions/upload-artifact from 2 to 3 (#262) 66777d9 Bump actions/download-artifact from 2 to 3 (#261) 14d4adb use new beaker-action for building test image (#265) af47287 Update pytorch-lightning requirement from <1.6,>=1.5 to >=1.5,<1.7 (#248) b1df9a4 use beaker-run action for GPU Tests (#263) 0a7468e fix release job (#260) c1b16b2 Bump furo from 2022.3.4 to 2022.4.7 (#259) b55aaf2 use beaker-py to submit GPU tests (#258) b2a93a9 Logging part 2: denoising run logging and making Dirk happy (#252) ff6be8d Update click requirement from <=8.0.4,>=7.0 to >=7.0,<8.1.3 (#254) 83d78cc Bump mypy from 0.941 to 0.942 (#243) 3769327 Bump sphinx from 4.4.0 to 4.5.0 (#245) 81fc5c5 Bump black from 21.12b0 to 22.3.0 (#246) e46059b Update tqdm requirement from <4.64,>=4.62 to >=4.62,<4.65 (#256) bbdeb6f Revert "Set $TEMP (#241)" b9fd9e9 Fix tracking dependencies between steps (#249) 53502e1 Pretty-print a step graph (#250) d5328c9 Fix dissimilar objects hashing to the same thing (#240) ccc37ce Autodetect debugger and turn off multicore (#251) 5c39f61 Pin click 5bb0fad Logging improvements (#233) 037e4a0 fix bug with FromParams (#242) e142530 Bump actions/cache from 2 to 3 (#236) 878402d Set $TEMP (#241) 2d9fa0c fix bug w/ TorchTrainStep working dir (#238) 410faeb Multicore Parallelism (#204) 9e8e99f Update datasets requirement from <2,>=1.12 to >=1.12,<3 (#234) 40e0a1a Bump mypy from 0.940 to 0.941 (#230) ede7428 add name to changelog workflow 4bb659b Bump actions/setup-python from 2 to 3 (#229) 5db1a6a Bump actions/checkout from 1 to 3 (#228) 8049104 Update torch version where it's hard-coded, add an automatic remind to do this stuff in the future (#227) fe05449 add back intersphinx inventory links for HF libraries (#222) 9927749 Bump mypy from 0.931 to 0.940 (#226) 29ab68b Update torch requirement from <1.11,>=1.9 to >=1.9,<1.12 (#225) a3fc83b Bump furo from 2022.2.23 to 2022.3.4 (#218) 28e839e Bump fairscale from 0.4.5 to 0.4.6 (#224) f18d393 Update tqdm requirement from <4.63,>=4.62 to >=4.62,<4.64 (#213) 54c4a8d automatically keep copyright up-to-date (#221) 06adb07 Allow setting the run name as a command-line option (#212) 71e0639 Update cached-path requirement from <1.1,>=1.0 to >=1.0,<1.2 (#217) 5d4660a Temporarily remove intersphinx links to HF docs (#220) 13c7f3f Merge pull request #216 from allenai/VersionForFromParams 0027cb2 Merge pull request #215 from allenai/fix-fully-qualified-name-recognition 76f9922 Add "Step.workspace" property (#210)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.7.0.tar.gz(133.97 KB)
ai2_tango-0.7.0-py3-none-any.whl(172.16 KB)
v0.6.0(Feb 25, 2022)
What's new

Added 🎉

New example that finetunes a pre-trained ResNet model on the Cats & Dogs dataset.

Added a '@requires_gpus' decorator for marking tests as needing GPUs. Tests marked with this will be run in the "GPU Tests" workflow on dual k80 GPUs via Beaker.

Added the "-w/--workspace" option to tango run and tango server commands. This option takes a path or URL, and instantiates the workspace from the URL using the newly added Workspace.from_url() method.

Added the "workspace" field to TangoGlobalSettings.

Added the "environment" field to TangoGlobalSettings for setting environment variables each time tango is run.

Added a utility function to get a StepGraph directly from a file.

Added tango.settings module and tango settings group of commands.

A format for storing sequences as SqliteSparseSequence

A way to massage kwargs before they determine the unique ID of a Step

Changed ⚠️

local_workspace.ExecutorMetadata renamed to StepExecutionMetadata and now saved as execution-metadata.json.

tango run without the option "-w/--workspace" or "-d/--workspace-dir" will now use a MemoryWorkspace instead of a LocalWorkspace in a temp directory, unless you've specified a default workspace in a TangoGlobalSettings file.

Moved tango.workspace.MemoryWorkspace and tango.local_workspace.LocalWorkspace to tango.workspaces.*.

Moved tango.step_cache.MemoryStepCache and tango.step_cache.LocalStepCache to tango.step_caches.*.

Deprecated the -d/--workspace-dir command-line option. Please use -w/--workspace instead.

Fixed ✅

Fixed a small bug LocalWorkspace would fail to capture the conda environment in our Docker image.

Fixed activation of FILE_FRIENDLY_LOGGING when set from the corresponding environment variable.

Fixed setting log level via the environment variable TANGO_LOG_LEVEL.

Use relative paths within the work_dir for symbolic links to the latest and the best checkpoints in TorchTrainStep.

Fixed some scenarios where Tango can hang after finishing all steps.

distributed_port and log_every parameters won't factor into TorchTrainStep's unique ID.

MappedSequence now works with slicing.

MappedSequence now works with Huggingface Dataset.

Uncacheable steps are now visible in Tango UI.

Fixed bug in Registrable.list_available() where an error might be raised if the default implementation hadn't been explicitly imported.

Fixed issue where having a default argument to the run() method wasn't getting applied to the step's unique ID.

Commits

f9da0af Merge pull request #211 from allenai/Massage e78dcbe Allow setting environment variables in tango settings, fix bug with TANGO_LOG_LEVEL env var (#209) 82404b6 Re-create LICENSE so GitHub will show it (#208) 0fadecf Bump furo from 2022.2.14.1 to 2022.2.23 (#207) 787b6e6 Merge pull request #206 from allenai/settings c3401f2 Merge pull request #205 from allenai/RobustnessFixes 7ceda9c Merge pull request #201 from allenai/workspace-prep 6dd7d86 Merge pull request #200 from allenai/uncacheable-steps-in-server 5ad3f44 Bump furo from 2022.1.2 to 2022.2.14.1 (#199) 3528230 Update filelock requirement from <3.5,>=3.4 to >=3.4,<3.7 (#202) 21d6d40 Merge pull request #193 from allenai/StepGraphFromFile 258a7d2 skip 'distributed_port' and 'log_every' in unique ID (#197) dd4c47f Merge pull request #192 from allenai/CloseSqliteHarder cc94e1c Merge pull request #156 from allenai/DocumentationRefresh 5cc86b8 Rename "ExecutorMetadata" -> "StepExecutionMetadata" (#195) 6aecab7 Bump myst-parser from 0.16.1 to 0.17.0 (#191) 6478293 make pushing test image to Beaker more robust (#190) 7c1ac5b Finetune resnet Example for Tango (#150) 5187b01 update docs for integration tests and gpu tests timeout 95b78b5 Add new manually triggered workflow for integration tests, other bug fixes (#188) 19f7b31 Merge pull request #189 from allenai/fix-checkpoint-path-link a438b26 Workflow quickfix 671a6dc verify exit code of beaker job (#187) 7ccad94 Merge pull request #186 from allenai/add-tests bf6ecd0 Run GPU tests on Beaker (#183)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.6.0.tar.gz(121.30 KB)
ai2_tango-0.6.0-py3-none-any.whl(157.12 KB)
v0.5.0(Feb 10, 2022)
What's new

Added 🎉

Added TrainingEngine abstraction to torch integration.

Added FairScale with a FairScaleTrainingEngine that leverages FairScale's FullyShardedDataParallel. This is meant to be used within the TorchTrainStep.

All PyTorch components (such as learning rate schedulers, optimizers, data collators, etc) from the transformers library and now registered under the corresponding class in the torch integration. For example, transformers Adafactor optimizer is registered as an Optimizer under the name "transformers::Adafactor". More details can be found in the documentation for the transformers integration.

Changed ⚠️

Various changes to the parameters othe TorchTrainStep due to the introduction of the TrainingEngine class.

Params logged as DEBUG level instead of INFO to reduce noise in logs.

The waiting message for FileLock is now clear about which file it's waiting for.

Added an easier way to get the default Tango global config

Most methods to TorchTrainCallback also take an epoch parameter now.

WandbTrainCallback now logs peak GPU memory occupied by PyTorch tensors per worker. This is useful because W&B's system metrics only display the total GPU memory reserved by PyTorch, which is always higher than the actual amount of GPU memory occupied by tensors. So these new metrics give a more accurate view into how much memory your training job is actually using.

Plain old Python functions can now be used in Lazy objects.

LocalWorkspace now creates a symlink to the outputs of the latest run.

Tango is now better at guessing when a step has died and should be re-run.

Tango is now more lenient about registering the same class under the same name twice.

When you use dict instead of Dict in your type annotations, you now get a legible error message. Same for List, Tuple, and Set.

Fixed ✅

Fixed a bug in Registrable and FromParams where registered function constructors would not properly construct arguments that were classes.

Fixed a bug in FromParams that would cause a crash when an argument to the constructor had the name params.

Made FromParams more efficient by only trying to parse the params as a Step when it looks like it actually could be a step.

Fixed bug where Executor would crash if git command could not be found.

Fixed bug where validation settings were not interpreted the right way by the torch trainer.

When you register the same name twice using Registrable, you get an error message. That error message now contains the correct class name.

Commits

a39a69f Merge pull request #161 from allenai/FromParamsDuJour 3063a92 CHANGELOG quick fix cd006ae Add TrainEngine abstraction to TorchTrainStep, add FairScale integration, improve transformers integration (#77) 93438eb Update setuptools requirement from <=59.5.0 to <60.8.0 (#170) e57dd91 Bump sphinx-copybutton from 0.4.0 to 0.5.0 (#174) a8b1bdc split Docker build into seperate workflow, only run when necessary (#178) 59c91f7 make install comments work on all shells (#179) a059416 Merge pull request #160 from allenai/GuessStepDirBetter de7195d more fixes for conda-forge (#177) 75e9d42 use conda in Docker image, multi-stage build (#172) 611e446 Merge pull request #176 from allenai/latest-outputs 7241d20 Merge pull request #175 from allenai/self-contained-tests 83aa692 Merge pull request #153 from allenai/LazyWithoutFromParams 893e601 use virtualenv within Docker (#167) 178b8bd Merge pull request #171 from allenai/LenientRegister 6c765c8 Merge pull request #169 from allenai/InformativeFileLock 91ff7ac Merge pull request #168 from allenai/DefaultGlobalConfig 5d602fb push Docker images to GHCR.io (#166) 2b26fc8 set 'resume' to 'allow' instead of 'auto' (#155) 26771e7 fix bug when git missing (#163) 9009119 Add Dockerfile (#162) a02155d Add a required flag to the README for gpt2-example (#159)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.5.0.tar.gz(114.58 KB)
ai2_tango-0.5.0-py3-none-any.whl(147.12 KB)
v0.4.0(Jan 27, 2022)
What's new

Changed ⚠️

Default log level is WARNING instead of ERROR.

The web UI now renders the step graph left-to-right.

The web UI now shows runs by date, with the most recent run at the top.

The web UI now shows steps in a color-coded way.

The --include-package flag now also accepts paths instead of module names.

Fixed ✅

Ensure tqdm log lines always make it into the log file out.log even when log level is WARNING or ERROR.

Commits

5ff51d6 Fix GPT2 example. (#158) 4011482 make --include-package accept paths (#157) 92b8fe5 Merge pull request #148 from allenai/RunsWithDates a4417e5 fix gpt2 config 797e3e8 minor logging tweaks (#145) 42654e6 Prepare for release v0.4.0rc5 df301ef Merge pull request #119 from allenai/RunGeneration 42535c9 Add TorchEvalStep to torch integration, use "device_count" in TorchTrainStep instead of "devices" (#120) ecc6087 Store log output to a file in run directory, other logging improvements (#132) 9008255 Merge pull request #141 from allenai/dependabot/pip/sphinx-4.4.0 8e09b66 Merge pull request #139 from allenai/remove-no-logging 26318b2 Merge pull request #133 from allenai/dependabot/pip/mypy-0.931 7c91c4b Merge pull request #126 from allenai/dependabot/pip/sphinx-4.3.2 a178076 Merge pull request #128 from allenai/dependabot/pip/mypy-0.930 106a8cf Merge pull request #130 from allenai/dependabot/pip/furo-2022.1.2 6db7b4c Merge pull request #129 from allenai/run-name-at-end b93d22d Add typing info to package (PEP 561) (#131) e8867ae fix StopEarlyCallback state recovery 465a525 Prepare for release v0.4.0rc4 7a65540 CHANGELOG quick fix 31622d1 add logo to docs and README (#121) b17d325 fix bug with StepInfo (#122) d5698a0 Bump myst-parser from 0.16.0 to 0.16.1 (#118) 69d1dc8 Bump mypy from 0.910 to 0.920 (#117) 9739944 Prepare for release v0.4.0rc3 20138ce improve release notes generation script 760b4f2 Add DatasetsFormat, making LoadDataset cacheable, fix bug with KeyboardInterrupt (#114) e51691f Improvements to W&B callback (#115) d044f6e Add pre/post epoch callbacks (#113) ae1ae0b Bump myst-parser from 0.15.2 to 0.16.0 (#111) c605a1e Merge pull request #90 from allenai/SqliteDictFormat 8d6804d Prepare for release v0.4.0rc2 f404541 Merge pull request #110 from allenai/Conda c154c92 Merge pull request #101 from allenai/Euler 288e02f ensure all integrations are imported if we can't find registered name (#109) e033170 Bump black from 21.11b1 to 21.12b0 (#102) b2781fd FAQ not FAQs :) (#108) 81e1225 add FAQs to docs (#107) 2148b74 Merge pull request #106 from allenai/Favicon f812b73 fix bug with resolving lazy step inputs (#105) 110fc79 Merge pull request #104 from allenai/why-tango eab62d8 fix bug in distributed training (#103) 901631f Removed scary warning e86b843 Better summary 3bfd7c1 adjust dependency pinning (#100) 72aaa53 Merge pull request #79 from allenai/jon/html-viz d45ec0c Merge pull request #99 from allenai/SkipGitTests 98b022b make sure workspaces can be imported from base module (#98) 07ac494 Merge pull request #97 from allenai/lower-click-pin 5ccefa9 fix prelease indicator condition in CI 04a5ab8 Prepare for release v0.4.0rc1 b2c09e3 fix typo in example (#96) aba5758 Merge pull request #94 from allenai/dependabot/pip/cached-path-gte-0.3.3-and-lt-1.1.0 48b0b24 Merge pull request #92 from allenai/dependabot/pip/datasets-gte-1.12-and-lt-1.17 0107672 Merge pull request #91 from allenai/dependabot/pip/sphinx-4.3.1 7d9d919 Merge pull request #93 from allenai/NoOverrides b5907de Merge pull request #67 from allenai/ResponsibleSteps 20951ea Bump furo from 2021.11.16 to 2021.11.23 (#89) 8bb00c4 Bump black from 21.11b0 to 21.11b1 (#88) f240ac4 update filelock + cached_path, improve release scripts (#87) eba4b8e Bump black from 21.10b0 to 21.11b0 (#86) bc80bb8 Merge pull request #85 from allenai/dependabot/pip/filelock-gte-3.3-and-lt-3.5 17d28c7 Bump furo from 2021.11.15 to 2021.11.16 (#84) 4118912 Merge pull request #82 from allenai/dependabot/pip/furo-2021.11.15 cb1b853 Merge pull request #62 from allenai/dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 30f7a13 W&B log as step+1 (#76) aab58b6 add some conda instructions to CONTRIBUTING.md (#81)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0.tar.gz(103.76 KB)
ai2_tango-0.4.0-py3-none-any.whl(131.77 KB)
v0.4.0rc5(Jan 20, 2022)
What's new

Added 🎉

Added TorchEvalStep to torch integration, registered as "torch::eval".

Changed ⚠️

Renamed aggregate_val_metric to auto_aggregate_val_metric in TorchTrainStep.

devices parameter to TorchTrainStep replaced with device_count: int.

Run name printed at the end of a run so it's easier to find.

Type information added to package data. See PEP 561 for more information.

A new integration, transformers, with two new steps for running seq2seq models.

Added logging_tqdm, if you don't want a progress bar, but you still want to see progress in the logs.

Added threaded_generator(), for wrapping generators so that they run in a separate thread from the generator's consumer.

Added a new example for evaluating the T0 model on XSum, a summarization task.

Added MappedSequence for functionally wrapping sequences.

Added TextFormat, in case you want to store the output of your steps in raw text instead of JSON.

Steps can now list arguments in SKIP_ID_ARGUMENTS to indicate that the argument should not affect a step's unique id. This is useful for arguments that affect the execution of a step, but not the output.

Step now implements __str__, so steps look pretty in the debugger.

Added DatasetCombineStep, a step that combines multiple datasets into one.

Added common.logging.initialize_worker_logging() function for configuring logging from worker processes/threads.

Logs from tango run ... will be written to a file called out.log in the run directory.

Fixed ✅

Fixed torch StopEarlyCallback state not being recovered properly on restarts.

Fixed file friendly logging by removing special styling characters.

Ensured exceptions captured in logs.

LocalWorkspace now works properly with uncacheable steps.

When a Tango run got killed hard, with kill -9, or because the machine lost power, LocalWorkspace would sometimes keep a step marked as "running", preventing further executions. This still happens sometimes, but it is now much less likely (and Tango gives you instructions for how to fix it).

To make all this happen, LocalWorkspace now saves step info in a Sqlite database. Unfortunately that means that the workspace format changes and existing workspace directories won't work properly with it.

Fixed premature cleanup of temporary directories when using MemoryWorkspace

Commits

df301ef Merge pull request #119 from allenai/RunGeneration 42535c9 Add TorchEvalStep to torch integration, use "device_count" in TorchTrainStep instead of "devices" (#120) ecc6087 Store log output to a file in run directory, other logging improvements (#132) 9008255 Merge pull request #141 from allenai/dependabot/pip/sphinx-4.4.0 8e09b66 Merge pull request #139 from allenai/remove-no-logging 26318b2 Merge pull request #133 from allenai/dependabot/pip/mypy-0.931 7c91c4b Merge pull request #126 from allenai/dependabot/pip/sphinx-4.3.2 a178076 Merge pull request #128 from allenai/dependabot/pip/mypy-0.930 106a8cf Merge pull request #130 from allenai/dependabot/pip/furo-2022.1.2 6db7b4c Merge pull request #129 from allenai/run-name-at-end b93d22d Add typing info to package (PEP 561) (#131) e8867ae fix StopEarlyCallback state recovery
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0rc5.tar.gz(101.76 KB)
ai2_tango-0.4.0rc5-py3-none-any.whl(129.69 KB)
v0.4.0rc4(Dec 20, 2021)
What's new

Fixed ✅

Fixed a bug where StepInfo fails to deserialize when error is an exception that can't be pickled.

Commits

7a65540 CHANGELOG quick fix 31622d1 add logo to docs and README (#121) b17d325 fix bug with StepInfo (#122) d5698a0 Bump myst-parser from 0.16.0 to 0.16.1 (#118) 69d1dc8 Bump mypy from 0.910 to 0.920 (#117)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0rc4.tar.gz(86.76 KB)
ai2_tango-0.4.0rc4-py3-none-any.whl(111.40 KB)
v0.4.0rc3(Dec 16, 2021)
What's new

Added 🎉

Added DatasetsFormat format and LoadStreamingDataset step to datasets integration.

SqliteDictFormat for datasets.

Added pre_epoch() and post_epoch() callback methods to PyTorch TrainCallback.

Changed ⚠️

LoadDataset step from datasets integration is now cacheable, using the DatasetsFormat format by default. But this only works with non-streaming datasets. For streaming datasets, you should use the LoadStreamingDataset step instead.

Fixed ✅

Fixed bug where KeyboardInterrupt exceptions were not handled properly by steps and workspaces.

WandbTrainCallback now will use part of the step's unique ID as the name for the W&B run by default, to make it easier to indentify which tango step corresponds to each run in W&B.

WandbTrainCallback will save the entire TrainConfig object to the W&B config.

Commits

20138ce improve release notes generation script 760b4f2 Add DatasetsFormat, making LoadDataset cacheable, fix bug with KeyboardInterrupt (#114) e51691f Improvements to W&B callback (#115) d044f6e Add pre/post epoch callbacks (#113) ae1ae0b Bump myst-parser from 0.15.2 to 0.16.0 (#111) c605a1e Merge pull request #90 from allenai/SqliteDictFormat
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0rc3.tar.gz(86.37 KB)
ai2_tango-0.4.0rc3-py3-none-any.whl(110.95 KB)
v0.4.0rc2(Dec 13, 2021)
What's new

Added 🎉

Sample experiment configurations that prove Euler's identity

Changed ⚠️

Loosened Click dependency to include v7.0.

Loosened datasets dependency.

Tightened petname dependency to exclude next major release for safety.

Fixed ✅

Workspace, MemoryWorkspace, and LocalWorkspace can now be imported directly from the tango base module.

Uncacheable leaf steps would never get executed. This is now fixed.

We were treating failed steps as if they were completed by accident.

The visualization had a problem with showing steps that never executed because a dependency failed.

Fixed a bug where Lazy inputs to a Step would fail to resolve arguments that come from the result of another step.

Fixed a bug in TorchTrainStep where some arguments for distributed training (devices, distributed_port) weren't being set properly.

Commits

f404541 Merge pull request #110 from allenai/Conda 4c347db Merge branch 'main' into Conda c154c92 Merge pull request #101 from allenai/Euler b3a8ae6 Revert "Make sure default steps are available when you run tango run" 76bda76 Merge remote-tracking branch 'origin/main' into Euler e073925 Revert "Import integrations safely" 5550ff8 Adds Conda to the readme 288e02f ensure all integrations are imported if we can't find registered name (#109) e033170 Bump black from 21.11b1 to 21.12b0 (#102) 2fd48cc Merge branch 'main' into Euler b2781fd FAQ not FAQs :) (#108) b31c8f2 Merge branch 'main' into Euler 81e1225 add FAQs to docs (#107) dec4a86 Merge remote-tracking branch 'origin/Euler' into Euler 23d7824 Brings back the Euler example 4327e22 Merge branch 'main' into Euler 2148b74 Merge pull request #106 from allenai/Favicon 5ce5c64 Merge branch 'main' into Favicon 18e556a Revert "Changelog" 97dd79d Merge branch 'main' into Euler f812b73 fix bug with resolving lazy step inputs (#105) 0a05047 Merge branch 'main' into Euler 89b315a Tango Favicon 110fc79 Merge pull request #104 from allenai/why-tango a67c617 Update README.md a5f5bd0 Merge branch 'main' into why-tango eab62d8 fix bug in distributed training (#103) e73a4a0 Update README.md eb3ef41 Merge branch 'main' into Euler c1c9bd0 Changelog 901631f Removed scary warning e86b843 Better summary cf9088d We no longer start the server during tests b257e46 Import integrations safely 10ec4f4 Pick the right mutable mapping a175370 Make sure default steps are available when you run tango run 1d4051a Make mypy happy 67e8a02 Moved complex arithmetic to https://github.com/allenai/tango-example 8a5c0d7 Adds test for steps that fail 3d551b3 Test for uncacheable leaf steps 21f0280 Don't explicitly run steps that don't need it 5fa4731 isort 9a3cd52 Changelog 6df6682 clarify comment c46e0b2 also pick any uncacheable direct dependencies of leaf steps c32a5cc Example steps for Euler's identity 1c97f10 Write step info for all steps before running any of them 3336a7e Reset step info when re-running a failed step 228b0f3 How did this slip by? f9f1d05 Make sure to execute uncacheable steps that are not a dependency of anything 3d44720 Add steps for complex arithmetic 3bfd7c1 adjust dependency pinning (#100) 72aaa53 Merge pull request #79 from allenai/jon/html-viz adea827 Merge branch 'jon/html-viz' of github.com:allenai/tango into jon/html-viz # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit. e771da7 make alert look nice d21a172 Merge branch 'main' into jon/html-viz d45ec0c Merge pull request #99 from allenai/SkipGitTests 28f17fa Don't check for git if we're not running in a repo 7db639e Merge branch 'main' into jon/html-viz 98b022b make sure workspaces can be imported from base module (#98) 07ac494 Merge pull request #97 from allenai/lower-click-pin f10b178 Merge branch 'main' into lower-click-pin 5ccefa9 fix prelease indicator condition in CI 9843c5a update CHANGELOG 91d6551 Merge branch 'main' into lower-click-pin 58dfd3e CHANGELOG.md 8a069f0 loosen Click requirement 85b7417 Shows an ugly but functional brief popup when you copy something 626f1b6 Merge branch 'main' into jon/html-viz b0cbf63 make tooltips meaningful since i cannot remove them 422ae0d Update tango/server/report.js 18869b7 Make local results work 3b1326a Python 3.7 again 5a282ff Warning to the future 2f54569 Give the option of tracking dependencies properly through Workspace f9f7bfb We don't have this method anymore. 77ae786 Silence a warning 0d532c5 Reformat setup.py only in Python 3.7? 4f1e786 Changelog b500f0e Merge branch 'main' into jon/html-viz 5c0b794 Merge branch 'main' into jon/html-viz 5c455e2 Include assets in package 4153e9e Fix import order 85dd1af Print a direct link to the run if we have one 01f0343 Makes it so you can run the server from any directory 300b722 Make frontend and backend consistent in their terminology 0ef1350 Fix some errors in the server bd07ad5 Merge remote-tracking branch 'origin/ResponsibleSteps' into jon/html-viz 0cde4e7 pr updates 7bf9488 Makes tests pass e96f6f3 Merge branch 'ResponsibleSteps' into jon/html-viz 8800bc9 Merge branch 'ResponsibleSteps' into jon/html-viz 378f649 pr gix 5047f6b add logo 4a7487b add in reloading every second a1131ed pr fixes 95f1cbe pr fixes d47b624 pr fixes 5962699 pr fixes a974369 fix merge conflicts 3280491 initial code to serve and display tango viz
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0rc2.tar.gz(84.24 KB)
ai2_tango-0.4.0rc2-py3-none-any.whl(108.28 KB)
v0.4.0rc1(Dec 1, 2021)
What's new

Added 🎉

Introduced the concept of the Workspace, with LocalWorkspace and MemoryWorkspace as initial implementations.

Added a stub of a webserver that will be able to visualize runs as they happen.

Added separate classes for LightningTrainingTypePlugin, LightningPrecisionPlugin, LightningClusterEnvironmentPlugin, LightningCheckpointPlugin for compatibility with pytorch-lightning>=1.5.0.

Removed 👋

Removed old LightningPlugin class

Removed requirement of the overrides package

Changed ⚠️

Made it possible to construct a step graph out of Step objects, instead of constructing it out of StepStub objects.

Removed dataset fingerprinting code, since we can now use Step to make sure things are cached.

Made steps deterministic by default.

Brought back MemoryStepCache, so we can run steps without configuring anything.

W&B torch::TrainCallback logs with step=step+1 now so that training curves in the W&B dashboard match up with checkpoints saved locally and are easier to read (e.g. step 10000 instead of 9999).

filelock >= 3.4 required, parameter poll_intervall to tango.common.file_lock.FileLock.acquire renamed to poll_interval.

Fixed ✅

Fixed bug in FromParams where a parameter to a FromParams class may not be instantiated correctly if it's a class with a generic type parameter.

Commits

b2c09e3 fix typo in example (#96) aba5758 Merge pull request #94 from allenai/dependabot/pip/cached-path-gte-0.3.3-and-lt-1.1.0 4ae6115 Update requirements.txt d4d0655 Merge branch 'main' into dependabot/pip/cached-path-gte-0.3.3-and-lt-1.1.0 48b0b24 Merge pull request #92 from allenai/dependabot/pip/datasets-gte-1.12-and-lt-1.17 6044f8b Update cached-path requirement from <0.4.0,>=0.3.3 to >=0.3.3,<1.1.0 1ae82aa Merge branch 'main' into dependabot/pip/datasets-gte-1.12-and-lt-1.17 0107672 Merge pull request #91 from allenai/dependabot/pip/sphinx-4.3.1 fa47e54 Merge branch 'main' into dependabot/pip/datasets-gte-1.12-and-lt-1.17 38c0b42 Merge branch 'main' into dependabot/pip/sphinx-4.3.1 7d9d919 Merge pull request #93 from allenai/NoOverrides dfa461c Removes the dependency on the overrides package ff18197 Update datasets requirement from <1.16,>=1.12 to >=1.12,<1.17 fc69dd0 Bump sphinx from 4.3.0 to 4.3.1 b5907de Merge pull request #67 from allenai/ResponsibleSteps 267a6e4 clean up config usage 82862ef Merge branch 'main' into ResponsibleSteps 20951ea Bump furo from 2021.11.16 to 2021.11.23 (#89) 8d8670a Optional server 03049fa Handle the log level consistently c620405 Merge branch 'main' into ResponsibleSteps 8bb00c4 Bump black from 21.11b0 to 21.11b1 (#88) cd5a70a Fix tests 71cfbd7 Don't cache uncacheable steps 7839fd8 Merge branch 'main' into ResponsibleSteps f240ac4 update filelock + cached_path, improve release scripts (#87) 967ecb2 Merge branch 'main' into ResponsibleSteps 10634aa Don't show inherited from_params 4b104be Fix test 4e9910d Avoid a naming conflict in computer science 6db2c29 Improved documentation 44f79ec Added blurb 7f68a97 Use enum for step states 19e6de2 Click logging is disabled by default, enabled in the CLI use case eba4b8e Bump black from 21.10b0 to 21.11b0 (#86) 1d80766 Log the start of a run 6190e2d Merge branch 'main' into ResponsibleSteps 3550c3f Cleaner workspace docs 08d4056 Better StepCache docs d5ede4e Formatting bc80bb8 Merge pull request #85 from allenai/dependabot/pip/filelock-gte-3.3-and-lt-3.5 d69bac4 Check whether a run name already exists cde0f14 Unused import a80a87f Fix the case where a step's cacheability changes across restarts de5f248 Improve comment f13d717 Losely pin petname 5d72923 Merge branch 'main' into ResponsibleSteps 914902b Merge branch 'ResponsibleSteps' of https://github.com/allenai/tango into ResponsibleSteps 2f5a266 Merge pull request #83 from allenai/petew-ResponsibleSteps b3df88f Update filelock requirement from <3.4,>=3.3 to >=3.3,<3.5 17d28c7 Bump furo from 2021.11.15 to 2021.11.16 (#84) 8f2b48e add failing test case 1fc9860 Merge remote-tracking branch 'origin/main' into ResponsibleSteps 4118912 Merge pull request #82 from allenai/dependabot/pip/furo-2021.11.15 d5eb968 Merge branch 'main' into dependabot/pip/furo-2021.11.15 abac28a Fix the shortcut for running all (many) checks c4009de Merge branch 'main' into ResponsibleSteps e364766 Merge pull request #73 from allenai/petew-ResponsibleSteps 8ae0e27 More doctests 6211025 Format docs better 5814724 Fix docs cb1b853 Merge pull request #62 from allenai/dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 50caada Update requirements.txt 9c79b97 Makes the docs build 3ecb952 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 55c5fae Bump furo from 2021.11.12 to 2021.11.15 83add7e Important fixes 301347d Fix tests cfe29de Create workdir when requested 7b67142 Bring back "needed by" a46b3e8 This wasn't meant to be checked in. af4cc3c Use click through the logger ee81446 Merge branch 'ResponsibleSteps' into petew-ResponsibleSteps 7247528 We don't need this TODO right now. 9b8446c Merge branch 'main' into ResponsibleSteps 30f7a13 W&B log as step+1 (#76) eab2b7f Merge pull request #74 from allenai/Workspaces d91302b Simplify! e605399 changelog 9f360da Bring back deterministic step randomness, without breaking random step names a2c5cd8 fix order of imports 02ad624 Merge branch 'dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0' of https://github.com/allenai/tango into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 7f78b0a remove comment 8f8cbeb separating different plugin types b139181 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 761f873 Merge branch 'ResponsibleSteps' into Workspaces 12a480a Merge branch 'ResponsibleSteps' into petew-ResponsibleSteps b7d4e88 merge main 63f6029 Merge branch 'main' into ResponsibleSteps aab58b6 add some conda instructions to CONTRIBUTING.md (#81) e7e5c5e Fix symlink creation b547472 Adds a command to keep a server running permanently 596c278 Not sure how this line got lost 06d7681 Fixes and cleanup 3cebc24 fix bug caused by random seed d145216 fix comment 067436d clean up a4ce577 Creates and uses the concept of a workspace, so that the server can consume it 110eb07 ci e811f47 fix tests 460dd87 executor fixes 837a454 handle generic non-FromParams classes 7fed5a6 fix merge conflicts 12ae8e9 Fixing the torch test 0c208e1 Remove stale comment b44a0a0 Fix doctest 3e4eb42 Makes det_hash consistent across Python versions f23b56c Executable documentation! 38388ca Formatting 6aaa01e Fix some documentation 57825a0 Fix docs cb7591d Order imports correctly 🙄 8caea06 Removing unused imports f7f5d2e Changelog again 05ca4dd Changelog da16b5a Merge branch 'main' into ResponsibleSteps 8c51b45 Make nested steps work for classes that aren't FromParams 083516b Remember which extra modules we imported 54e7dbc We can't restore the registry like that. 455b756 Refactors the test to fail in new and exciting ways 62cd1d2 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 cd68752 Don't need this comment anymore 76c10f1 Remove fingerprint stuff a1f1af8 Merge branch 'FixImport' into ResponsibleSteps ed9ebc9 Fix Import 14bc2ee Merge branch 'main' into ResponsibleSteps 4a4ab58 Merge branch 'main' into dependabot/pip/pytorch-lightning-gte-1.4.0-and-lt-1.6.0 e5ea1af Fix after merge 7b92dd9 Merge remote-tracking branch 'origin/main' into ResponsibleSteps d066768 We were not actually using this function df07690 Mypy inspired changes 87ee96f Formatting 3823f98 🤦🏼 d018838 Fix executor test a7fb4ea Fix executor 0b1f310 Quiet, you 449a70d Fixes circular references 4fc1f34 Actually write the circular reference test 0df3538 Throw the error for the right reason afb3b61 WithUnresolvedSteps 1be6816 Update pytorch-lightning requirement 3c4cb19 Dicts are iterable, so these have to be swapped efc84d4 Adds new failing test 1f2bf16 Relative imports don't work 430112f Fix bug in test c3dd08d Add test that fails 118f8c2 Better name for the test a6ac3c9 Detect unsatisfiable dependencies 8eb7a48 Bring back parsing everything as a Step first :-/ bc6f886 Type checks 00b2df6 Makes the test pass a6ba11a Make the code more compatible with the IDE a611d48 We don't need these anymore 132153d Formatting d019499 Start fixing hard tests f4d77c1 Fix trivial tests fa6de92 Make steps responsible for their own execution cfe7007 Slightly more readable error message c0ded32 Typo
Source code(tar.gz)
Source code(zip)
ai2-tango-0.4.0rc1.tar.gz(73.97 KB)
ai2_tango-0.4.0rc1-py3-none-any.whl(96.08 KB)
v0.3.6(Nov 12, 2021)
What's new

Added 🎉

Added a .log_batch() method on torch::TrainCallback which is given the average loss across distributed workers, but only called every log_every steps.

Removed 👋

Removed .pre_log_batch() method on torch::TrainCallback.

Fixed ✅

Fixed typo in parameter name remove_stale_checkpoints in TorchTrainStep (previously was remove_state_checkpoints).

Fixed bug in FromParams that would cause failures when from __future__ import annotations was used with Python older than 3.10. See PEP 563 for details.

Commits

6b5cb24 support for PEP 563 in older Python versions (#80) 27657d9 Bump furo from 2021.10.9 to 2021.11.12 (#78) c8ac858 Bump sphinx from 4.2.0 to 4.3.0 (#75) 5516244 refactor TorchTrainStep (#70) b13bba7 Bump isort from 5.10.0 to 5.10.1 (#71)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.6.tar.gz(67.20 KB)
ai2_tango-0.3.6-py3-none-any.whl(87.96 KB)
v0.3.5(Nov 5, 2021)
What's new

Fixed ✅

Fixed a bug in FromParams where the "type" parameter was ignored in some cases where the Registrable base class did not directly inherit from Registrable.

Commits

5bdad24 Merge pull request #69 from allenai/weird-mix-fix 5434e23 test case e1108a5 CHANGELOG.md 2bbfa6f fix weird FromParams/Registrable bug
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.5.tar.gz(66.55 KB)
ai2_tango-0.3.5-py3-none-any.whl(86.54 KB)
v0.3.4(Nov 4, 2021)
What's new

Added 🎉

Added StopEarlyCallback, a TorchTrainCallback for early stopping.

Added parameter remove_stale_checkpoints to TorchTrainStep.

Changed ⚠️

Minor changes to TorchTrainCallback interface.

Weights & Biases TorchTrainCallback now logs best validation metric score.

Commits

c126442 torch train improvements (#64)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.4.tar.gz(66.52 KB)
ai2_tango-0.3.4-py3-none-any.whl(86.50 KB)
v0.3.3(Nov 4, 2021)
What's new

Added 🎉

Added support for PEP 604 in FromParams, i.e. writing union types as "X | Y" instead of "Union[X, Y]".

[internals] Added a spot for miscellaneous end-to-end integration tests (not to be confused with "tests of integrations") in tests/end_to_end/.

[internals] Core tests now run on all officially supported Python versions.

Fixed ✅

Fixed a bug in FromParams where non-FromParams class parameters were not instantiated properly (or at all).

Fixed a bug in FromParams where kwargs were not passed on from a wrapper class to the wrapped class.

Fixed small bug where some errors from git would be printed when executor metadata is created outside of a git repository.

Commits

ea6d2e5 another FromParams fix (#66) eeb1560 Update datasets requirement from <1.15,>=1.12 to >=1.12,<1.16 (#60) fe6dbe0 Bump isort from 5.9.3 to 5.10.0 (#61) 9f302b3 PEP 604 support (#59) 634bd71 Merge pull request #63 from allenai/StepGraphTests 9f513d6 Rename step_graph.py to step_graph_test.py 604001f tweak checkpointing/validate step 0920349 fix bug with git metadata (#56) af1b438 fix another FromParams bug, add spot for miscellaneous end-to-end tests (#58)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.3.tar.gz(65.86 KB)
ai2_tango-0.3.3-py3-none-any.whl(85.70 KB)
v0.3.2(Nov 2, 2021)
What's new

Fixed ✅

Fixed a bug with FromParams that caused .from_params() to fail when the params contained an object that was already instantiated.

tango command no longer installs a SIGTERM handler, which fixes some bugs with integrations that use multiprocessing.

Commits

ab47e21 Merge pull request #55 from allenai/no-sigterm-handler b759c46 remove sigterm handler c4e96cb fix example a1f9ec5 fix FromParams bug (#54) d5ef0ae Bump black from 21.9b0 to 21.10b0 (#53) b20f5b2 fix typo
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.2.tar.gz(65.72 KB)
ai2_tango-0.3.2-py3-none-any.whl(85.57 KB)
v0.3.1(Oct 29, 2021)
What's new

Changed ⚠️

Updated the LightningTrainStep to optionally take in a LightningDataModule as input.

Commits

4d77160 Merge pull request #52 from allenai/fix-typo 92874c2 fix release docs 26e36e4 Merge pull request #51 from allenai/lightning-data-module b10d256 update changelog 6d068a4 fix import order 1744f55 adding option for data module
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.1.tar.gz(65.72 KB)
ai2_tango-0.3.1-py3-none-any.whl(85.58 KB)
v0.3.0(Oct 28, 2021)
What's new

Added 🎉

Added IterableDatasetDict, a version of DatasetDict for streaming-like datasets.

Added a PyTorch Lightning integration with LightningTrainStep.

Fixed ✅

Fixed bug with FromParams and Lazy where extra arguments would sometimes be passed down through to a Lazy class when they shouldn't.

Commits

a95dbae fix bugs with initializing lightning loggers and plugins (#50) 7e9c354 fix bug w/ DataLoader and PTL d634ea8 add isort (#49) 277b0e2 add IterableDatasetDict (#46) db06e70 Merge pull request #45 from allenai/pytorch-lightning 6174381 fix CHANGELOG 501bf73 fix failing test 686be74 Merge branch 'main' into pytorch-lightning 4f3f328 add torch:: to torch integrations 293fac1 doc and general fixes 2b2a6a3 Update tango/integrations/pytorch_lightning/init.py d46a0ba Update docs/source/api/integrations/pytorch_lightning.rst 3aeb825 fix docs 375ff58 update docs c6a1e38 update ci 8110242 PyTorch Lightning Integration 60121c0 update docs, print unicode characters by name (#44) 7beab21 clean up docs 2f6871b only print ascii characters (#43)
Source code(tar.gz)
Source code(zip)
ai2-tango-0.3.0.tar.gz(65.37 KB)
ai2_tango-0.3.0-py3-none-any.whl(84.85 KB)
v0.2.4(Oct 22, 2021)
What's new

Added 🎉

Added support for torch 1.10.0.

Changed ⚠️

--file-friendly-logging flag is now an option to the main tango command, so needs to be passed before run, e.g. tango --file-friendly-logging run ....

Fixed ✅

Fixed bug with Step.from_params.

Ensure logging is initialized is spawn processes during distributed training with TorchTrainStep.

Commits

d497e7a Update torch requirement from <1.10.0,>=1.9.0 to >=1.9.0,<1.11.0 (#42) fbad9b2 fix failing test 409b50a ensure logging initialize in spawn distributed workers 1714390 move file_friendly_logging flag back to main command af8bc69 fix bug in Step.from_params 4e5b406 add TangoMetadata to docs
Source code(tar.gz)
Source code(zip)
ai2-tango-0.2.4.tar.gz(62.56 KB)
ai2_tango-0.2.4-py3-none-any.whl(77.26 KB)

A Python library for choreographing your machine learning research.

Related tags

Overview

Quick links

Installation

Installing with pip

Installing from source

Team

License

Comments

🐛 Describe the bug

Versions

🐛 Describe the bug

Versions

Tips for reviewers

Before submitting

After submitting

Review items to complete

🐛 Describe the bug

Versions

🚀 The feature, motivation and pitch

Alternatives

Additional context

Dropped Python3.6 support

[13.0.0] - 2022-12-30

Fixed

Changed

Added

[13.0.0] - 2022-12-30

Fixed

Changed

Added

[12.6.0] - 2022-10-02

Added

Fixed

Changed

[12.5.2] - 2022-07-18

Added

v6.0.0

v6.0.0b2

v6.0.0b1

Release 6.0.0 (released Dec 29, 2022)

Dependencies

Incompatible changes

Features added

5.11.4

Changes

:package: Dependencies

5.11.3

Changes

:beetle: Fixes

:construction_worker: Continuous Integration

v5.11.3

Changes

:beetle: Fixes

:construction_worker: Continuous Integration

5.11.2

Changes

5.11.1

Changes December 12 2022

5.11.4 December 21 2022

5.11.3 December 16 2022

5.11.2 December 12 2022

5.11.1 December 12 2022

5.11.0 December 12 2022

Before submitting

After submitting

22.12.0

Preview style

Configuration

Parser

Integrations

22.12.0

Preview style

Configuration

Parser

Integrations

Releases(v1.1.0)

v1.1.0(Dec 1, 2022)

What's new

Installing with `pip`