As pointed out in discussion in #1691, we should reconsider gc implementation. Currently, if called without any options, dvc will collect current branch dependencies and outputs checksums, and remove everything besides it. We can easily clear history of changes with this command. gc should be safer with default options. Straightforward implementation could get all outputs for all revisions in git repo and remove everything that is not on list.

As pointed out by @Suor, this approach might be slow for repository with long history.

Currently dvc metrics show can show metric values across different branches (-a) and different tags (-T). Can you consider supporting showing different metric values across different commits in the same branch?

The background of this is (simplified example): say I'm currently training a model, where I'm changing a certain parameter, param1 (for instance, number of trees in a forest). The way I probably would like to work is to find a first value for param1, commit the current state, continue changing param1 and continue committing the successive states that I consider worth saving. At some point I would like to look back and identify the setup that gave me the best results.

The way DVC currently works forces me to create a new branch/tag for each trial I want to keep track of, and this seems a bit overwhelming.

Depending on how different the experiments I'm running are and their level of granularity I could decide how to keep track of them (new commits VS new branches/tags).

Notes:

• The example above is overly simplified and there are better ways of tuning specific models parameters. But this gets more complicated if I'm changing more stuff (model hyperparameters, data processing, features to use, etc).
• If dvc were to support what I'm proposing here, an extra argument would probably be required to limit how many commits DVC would look back at. Otherwise it would show all the metric values since the beginning of the repo history, which can be unhelpful and messy.

Everytime that im trying to add some individuals files or complete directories the same unexpected error appears:

> dvc add -v -R model
DEBUG: Trying to spawn '['c:\\users\\luisfelipe_melo_mora\\appdata\\local\\programs\\python\\python37-32\\python.exe', 'C:\\Users\\luisfelipe_melo_mora\\AppData\\Local\\Programs\\Python\\Python37-32\\Scripts\\dvc', 'daemon', '-q', 'updater']'
DEBUG: Spawned '['c:\\users\\luisfelipe_melo_mora\\appdata\\local\\programs\\python\\python37-32\\python.exe', 'C:\\Users\\luisfelipe_melo_mora\\AppData\\Local\\Programs\\Python\\Python37-32\\Scripts\\dvc', 'daemon', '-q',
'updater']'
ERROR: unexpected error - Already unlocked
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\dvc\main.py", line 48, in main
    cmd = args.func(args)
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\dvc\command\base.py", line 48, in __init__
    updater.check()
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\dvc\updater.py", line 54, in check
    self._with_lock(self._check, "checking")
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\dvc\updater.py", line 45, in _with_lock
    func()
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\flufl\lock\_lockfile.py", line 338, in __exit__
    self.unlock()
  File "c:\users\luisfelipe_melo_mora\appdata\local\programs\python\python37-32\lib\site-packages\flufl\lock\_lockfile.py", line 287, in unlock
    raise NotLockedError('Already unlocked')
flufl.lock._lockfile.NotLockedError: Already unlocked
------------------------------------------------------------


Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

I have a remote configuration by SSH:

['remote "myssh"']
url = ssh://domain:/path
user = myuser
port = 22
ask_password = true
[core]
remote = myssh

And here the version of dvc that im using:

> dvc version
DVC version: 0.69.0
Python version: 3.7.4
Platform: Windows-10-10.0.17134-SP0
Binary: False
Package: pip
Cache: reflink - False, hardlink - True, symlink - False

Thanks for your help!

UPDATE: Skip to https://github.com/iterative/dvc/issues/2799#issuecomment-650464000 for a summary and updated requirements, and https://github.com/iterative/dvc/issues/2799#issuecomment-652969635 for the beginning of the implementation discussion.

Problem

There are a lot of discussions on how to manage ML experiments with DVC. Today's DVC design allows ML experiments through Git-based primitives such as commits and branches. This works nicely for large ML experiments when code writing and testing required. However, this model is too heavy for the hyperparameters tuning stage when the user makes dozens of small, one-line changes in config or code. Users don't want to have dozens of Git-commits or branches.

Requirements

A lightweight abstraction needs to be created in DVC to support hyperparameters-like tiny experiments without Git-commits. Hyperparameters tunning stage can be considered as a separate user activity outside of Git workflow. But the result of this activity still needs to be managed by Git preferably by a single commit.

High-level requirements to the hyperparameters tunning stage:

1. Run. Run dozens of experiments without committing any results into Git while keeping track of all the experiments. Each of the experiments includes a small config change or code change (usually, 1-2 lines).
2. Compare. A user should be able to compare two experiments: see diffs for code (and probably metrics)
3. Visualize. A user should be able to see all the experiments results: metrics that were generated. It might be some table with metrics or a graph. CSV table needs to be supported for custom visualization.
4. Propagate. Choose "the best" experiment (not necessarily the highest metrics) and propagate it to the workspace (bring all the config and code changes. Important: without retraining). Then it can be committed to Git. This is the final result of the current hyperparameter tunning stage. After that, the user can continue to work with a project in a regular Git workflow.
5. Store. Some (or all) of the experiments might be still useful (in additional to "the best" one). A user should be able to commit them to the Git as well. Preferably in a single commit to keep the Git history clean.
6. Clean. Not useful experiments should be removed with all the code and data artifacts that were created. A special subcommand of dvc gc might be needed.
7. [*] Parallel. In some cases, the experiments can be run in parallel which aligns with DVC parallel execution plans: #2212, #755. This might not be implemented now (in the 1st version of this feature) but it is important to support parallel execution by this new lightweight abstraction.
8. Group. Iterations of hyperparameters tuning might be not related to each other and need to be managed and visualized separately. Experiments need to be grouped somehow.

What should NOT be covered by this feature?

This feature is NOT about the hyperparameter grid-search. In most cases, hyperparameters tuning is done by users manually using "smart" assumptions and hypotheses about hyperparameter space. Grid-search can be implemented on top of this feature/command using bash for example.

9. The ability to run the experiments from bash might be also a requirement for this feature request.

Possible implementations

This is an open question but many data scientists create directories for each of the experiments. In some cases, people create directories for a group of experiments and then experiments inside. We can use some of these ideas/practices to better align with users' experience and intuition.

Actions

This is a high-level feature request (epic). The requirements and an initial design need to be discussed and more feature requests need to be created. @iterative/engineering please share your feedback. Is something missing here?

EDITED:

Related issues

#2379 https://github.com/iterative/dvc/issues/2532 #1018 can be relevant (?) Discussion

For an ML experiment, it is important to know metrics as well as the parameters that were used in order to get the metrics. Today there is no training/processing parameter concept in DVC which creates a problem when a user needs to visualize an experiment for example in some UI.

A common workaround is to track parameters as metrics. However, the meaning of metrics is different. All the UI tools (including dvc metrics diff) need to show deltas where deltas do not make sense to some types of params. For example, delta for learning rate might be ok to see (values are still better), but delta for a number of layers (32, 64 or 128) does not make sense, the same for not numeric params like strings.

Also, config/parameters are a pre-requisite for experiment management (#2799 or CI/CD scenarios) when DVC (or other automation tools) need to change training regarding provided parameters.

Another benefit of the "understanding" parameter - DVC can use this information during repro. For example, DVC can realize that a step process which depends on config file config.json should not be run despite the config file change because the metrics it uses were not changed.

We need to introduce the experiment config file/parameters file with a fixed structure that DVC can understand.

Open questions:

1. Filename. config.json, dvcconfig.json, params.json.
2. File format: json, text config/ini (name=value), Hydra, ... We can run a survey.
3. How to track param dependency for stages. We can introduce a new type of dependency: param. If it is given then the stage depends on the file and on particular params values. Like dvc run -p learning_rate -p LL5_levels ....
4. DVC should probably support groups of params. Param name pattern could be used : dvc run -p 'processing.*' ...

I understand the merits of having multiple .dvc files for complex processes, but it would be just great to have the option to store the whole DAG in one Dvcfile!

I feel it might help the overall readability of the structure

I am dealing with a large hierarchical data set. One where artifacts are pulled from various directories to generate contiguous data sets that are then fed to ML processes downstream. I don't want to use dvc to reproduce the pipeline, at least not yet. My needs are rather to be able to version the overall image dataset hierarchy, for the purpose of manual inspection of the whole hierarchy and moving images into groups or removing them altogether when necessary.

This enables folks with less ML expertise control the data set they want to build by grouping the content together that they want to pick up when generating the data set. The data set is not a list of images, rather it is a list of lower dimensional feature vectors extracted from those images.

I'm finding dvc taking a potentially unreasonable amount of time to just add and commit. Perhaps I don't understand what I'm doing or haven't set my expectations correctly.

I wanted to keep these operations small in order to ensure things were working well. I have done the following. I have approximately 300K in total in this set right now.

1. store 60K images on local file system, under the data/ directory.
2. dvc add data/
3. dvc push -r remote. I forgot to commit here since things took so long and I wanted to see if pushing worked.
4. store 120K additional images to another sub directory under the data/ directory.
5. dvc add data/ -> goes through all of the files in data/ regardless. I ran -v here and showed the previous files.
6. dvc push -r remote.
7. dvc commit. Here dvc is taking the greater amount of 99% of system memory (13 GB) and appears to be causing disk thrashing. It's been running nearly for a day so far.

I am just looking for some guidance in managing a dataset of this nature using dvc in a way that will not eat up so much time, disk, compute, etc. If I'm doing something suboptimal, then I want to shine some light on that.

Follow up to https://github.com/iterative/dvc/pull/5198#issuecomment-774299750, #5301, and https://github.com/iterative/dvc.org/pull/2172#discussion_r573963049:

Question

add --to-remote is a bit strange because normally add doesn't move target data, rather tracks it in-place (analog to git add). But --to-remote implies that external data will be moved into the workspace at some point, which we skip for now but "pre-push" (transfer) it to remote storage (for later pull/fetch).

As of now add --to-remote has a similar result to get-url + add + push + remove, gc. So OK, maybe it's nice to have a shortcut to all that, but we already have import-url (--to-remote) to achieve the same.

The only difference vs. importing is that the data source is not recorded as a dependency in the .dvc file. So you can't update it or unfreeze+repro it. However I don't see any use cases where you would want to prevent the .dvc from having this dep, as you can simply never update or unfreeze it.

TLDR: I think import-url --to-remote is enough and what we should recommend for these situations. And add --to-remote breaks the Git analogy. Cc @dberenbaum

Improvement

• [x] But if we keep it, an improvement would be to NOT require the --external flag with it (cc @isidentical). This saves the user from typing a flag that is always needed, but also make sense since the data is not actually being treated as external in the sense that it won't be tracked/controlled in it's original location (requiring external cache, etc.).

• [x] Finish or close iterative/dvc.org/pull/2172 when this is decided.

Especially useful for "browsing" external DVC projects on Git hosting before using dvc get or dvc import. Looking at the Git repo doesn't show the artifacts because they're only referenced in DVC-files (which can be found anywhere), not tracked by Git.

Perhaps dvc list or dvc artifacts? (Or/and both dvc get list and dvc import list)

As mentioned in https://github.com/iterative/dvc.org/pull/611#discussion_r324998285 and other discussions.

UPDATE: Proposed spec (from https://github.com/iterative/dvc/issues/2509#issuecomment-533019513):

usage: dvc list [-h] [-q | -v] [--recursive [LEVEL]] [--rev REV | --versions]
                url [target [target ...]]

positional arguments:
  url         URL of Git repository with DVC project to download from.
  target      Paths to DVC-files or directories within the repository to list outputs
              for.

UPDATE: Don't forget to update docs AND tab completion scripts when this is implemented.

It seems like it is only possible to replace a dataset entirely and then re-run the analysis. Incremental processing would enable more efficient processing by avoiding recomputation. Here's how Pachyderm does it.

Uses Stdlib's pydoc to draw the output in the interactive mode while doing e.g. dvc pipeline show ...

Fixes #2807

• [x] ❗ Have you followed the guidelines in the Contributing to DVC list?

• [x] 📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.

• [x] ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addresses. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

Related MR: https://github.com/iterative/dvc.org/pull/831

Bug Report

Description

It looks like cloud versioning is failing to push when stage outputs are marked as cache: false. Might be related to https://github.com/iterative/dvc/issues/4428.

Reproduce

Set up a cloud-versioned remote for https://github.com/iterative/example-get-started and push to it.

Here's the output:

$ dvc push -v
2023-01-04 15:09:02,716 DEBUG: indexing latest worktree for 'dave-sandbox-versioning/example-get-started/remote'
2023-01-04 15:09:03,269 DEBUG: Pushing worktree changes to 'dave-sandbox-versioning/example-get-started/remote'
2023-01-04 15:09:03,658 ERROR: unexpected error - ('eval', 'live', 'plots')
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/Users/dave/Code/dvc/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/Users/dave/Code/dvc/dvc/commands/data_sync.py", line 59, in run
    processed_files_count = self.repo.push(
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/push.py", line 50, in push
    pushed += _push_worktree(
  File "/Users/dave/Code/dvc/dvc/repo/push.py", line 117, in _push_worktree
    return push_worktree(repo, remote, targets=targets, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/worktree.py", line 148, in push_worktree
    _update_out_meta(out, repo.index.data[workspace])
  File "/Users/dave/Code/dvc/dvc/repo/worktree.py", line 179, in _update_out_meta
    entry = index[key]
  File "/Users/dave/Code/dvc-data/src/dvc_data/index/index.py", line 179, in __getitem__
    return self._trie[key]
  File "/Users/dave/miniforge3/envs/dvc/lib/python3.10/site-packages/pygtrie.py", line 859, in __getitem__
    node, _ = self._get_node(key_or_slice)
  File "/Users/dave/miniforge3/envs/dvc/lib/python3.10/site-packages/pygtrie.py", line 552, in _get_node
    raise KeyError(key)
KeyError: ('eval', 'live', 'plots')
------------------------------------------------------------
2023-01-04 15:09:04,223 DEBUG: Removing '/private/tmp/.C898YREazrnPXGYsWbFsib.tmp'
2023-01-04 15:09:04,223 DEBUG: Removing '/private/tmp/.C898YREazrnPXGYsWbFsib.tmp'
2023-01-04 15:09:04,223 DEBUG: Removing '/private/tmp/.C898YREazrnPXGYsWbFsib.tmp'
2023-01-04 15:09:04,223 DEBUG: Removing '/private/tmp/example-get-started/.dvc/cache/.DGQMVLCJsyvDKpKuGC3DdL.tmp'
2023-01-04 15:09:04,225 DEBUG: Version info for developers:
DVC version: 2.38.2.dev23+ga24c38967.d20230104
---------------------------------
Platform: Python 3.10.2 on macOS-13.1-arm64-arm-64bit
Subprojects:
        dvc_data = 0.28.5.dev1+ge0d19ab
        dvc_objects = 0.14.0
        dvc_render = 0.0.17
        dvc_task = 0.1.9
        dvclive = 1.3.1
        scmrepo = 0.1.5
Supports:
        azure (adlfs = 2022.9.1, knack = 0.9.0, azure-identity = 1.7.1),
        gdrive (pydrive2 = 1.15.0),
        gs (gcsfs = 2022.11.0),
        hdfs (fsspec = 2022.11.0+18.g0c55724.dirty, pyarrow = 7.0.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        oss (ossfs = 2021.8.0),
        s3 (s3fs = 2022.11.0+6.g804057f, boto3 = 1.24.59),
        ssh (sshfs = 2022.6.0),
        webdav (webdav4 = 0.9.4),
        webdavs (webdav4 = 0.9.4),
        webhdfs (fsspec = 2022.11.0+18.g0c55724.dirty)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk3s1s1
Caches: local
Remotes: https, s3
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-01-04 15:09:04,225 DEBUG: Analytics is disabled.

Bug Report

Description

I am trying to track existing data from a storage account in Azure following current documentation.

Reproduce

1. dvc init
2. dvc remote add azcore azure://core-container
3. dvc remote add azdata azure://data-container
4. dvc add --external remote://azdata/existing-data

Expected

I'm not sure what is expected but the output is:

ERROR: unexpected error - : 'azure'

Environment information

Output of dvc doctor:

DVC version: 2.38.1 (pip)
---------------------------------
Platform: Python 3.9.6 on macOS-13.1-x86_64-i386-64bit
Subprojects:
	dvc_data = 0.28.4
	dvc_objects = 0.14.0
	dvc_render = 0.0.15
	dvc_task = 0.1.8
	dvclive = 1.3.1
	scmrepo = 0.1.4
Supports:
	azure (adlfs = 2022.11.2, knack = 0.10.1, azure-identity = 1.12.0),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s5s1
Caches: local
Remotes: azure, azure
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git

Additional Information:

2023-01-04 18:58:46,616 ERROR: unexpected error - : 'azure'
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/odbmgr.py", line 65, in __getattr__
    return self._odb[name]
KeyError: 'azure'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/commands/add.py", line 53, in run
    self.repo.add(
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/utils/collections.py", line 164, in inner
    result = func(*ba.args, **ba.kwargs)
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/repo/scm_context.py", line 156, in run
    return method(repo, *args, **kw)
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/repo/add.py", line 190, in add
    stage.save(merge_versioned=True)
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/stage/__init__.py", line 469, in save
    self.save_outs(
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/stage/__init__.py", line 512, in save_outs
    out.save()
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/output.py", line 643, in save
    self.odb,
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/output.py", line 450, in odb
    odb = getattr(self.repo.odb, odb_name)
  File "/Users/rmllopes/dev/auto-document-validation-ai/.venv/lib/python3.9/site-packages/dvc/odbmgr.py", line 67, in __getattr__
    raise AttributeError from exc
AttributeError
------------------------------------------------------------
2023-01-04 18:58:46,711 DEBUG: Version info for developers:
DVC version: 2.38.1 (pip)
---------------------------------
Platform: Python 3.9.6 on macOS-13.1-x86_64-i386-64bit
Subprojects:
	dvc_data = 0.28.4
	dvc_objects = 0.14.0
	dvc_render = 0.0.15
	dvc_task = 0.1.8
	dvclive = 1.3.1
	scmrepo = 0.1.4
Supports:
	azure (adlfs = 2022.11.2, knack = 0.10.1, azure-identity = 1.12.0),
	http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: azure, azure
Workspace directory: apfs on /dev/disk1s5s1
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-01-04 18:58:46,714 DEBUG: Analytics is enabled.
2023-01-04 18:58:46,911 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/st/05s6bkj55r9cw3hbrrdfvfqh0000gp/T/tmpoxhcmxev']'
2023-01-04 18:58:46,913 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/st/05s6bkj55r9cw3hbrrdfvfqh0000gp/T/tmpoxhcmxev']'

Bug Report

Description

S3 external outputs are broken for pipelines since https://github.com/iterative/dvc/commit/7211bd02eda74d5434f1b7996647f7027e6e83b0 because of a bug in s3fs (and probably in other filesystems). They will only break if running a stage for which an output doesn't already exist. When initializing the stage, DVC will try to remove the nonexistent output and raise a FileNotFound error.

Reproduce

dvc repro will break if there is an external output and that output does not exist yet.

In a new repo, using some <s3_path> that doesn't exist yet, do this:

$ echo 'foo' > foo
$ dvc stage add --external -n foo -d foo -O <s3_path> 'aws s3 cp params.yaml <s3_path>'
$ dvc repro -v

Expected

dvc repro shouldn't fail while removing outputs. In this case, it fails because of what seems like a bug or at least inconsistent behavior in fsspec. Like mentioned in https://github.com/iterative/dvc/issues/5961#issuecomment-1365822275, output.remove for s3fs and other async filesystems calls _expand_path. When the path doesn't exist and recursive=True, _expand_path raises FileNotFoundError. When recursive=False, it returns the path. It also returns the path for the LocalFileSystem regardless of whether recursive=True, so not sure if it was intended to raise an error only for this specific scenario.

Seems like the behavior between --no-download and --to-remote is inconsistent. We can fix in this PR or follow up with another one. For --to-remote, the outs metadata is updated with the new info but the workspace remains untouched, while --no-download drop the outs metadata and deletes anything in the workspace.

Originally posted by @dberenbaum in https://github.com/iterative/dvc/issues/8752#issuecomment-1369173097

I'm trying to version control my 210 G data which contains 2.41M files. When I run

dvc -v add data_clean/                                                        
Adding...

It stuck here for 20 hours. Is it supposed to happen?

My DVC repository is present in the GCE instance.

Thanks