Add to your project To enable lazydata, run in project root:

$ lazydata init

This resulted in:

'lazydata' is not recognized as an internal or external command, operable program or batch file.

on windows 10.

Useful for users that do not want to rely on Amazon S3 while using this package.

I'm running a Minio Server for storage, which mimics an S3 container.

boto3 (api doc here) supports custom endpoints that it's going to hit via the S3 API.

I would be good to add tests to this behaviour, along with testing pulls and pushes for normal behaviour. Using mocks maybe ?

EDIT: I also corrected a line which caused the package not to work for python 3.5, see commit 0d4f8fc

I noticed when the python application that is tracking some data crashes at some point, it can leave behind a corrupted yaml file. It is not that uncommon when you are in an exploratory phase of building an ML model to write a code that can crash for example because of memory issues etc. It would be great if yaml file handle closes after each track call to ensure the file does not get corrupted! Thanks! I really like this project!

I tested this out by creating and tracking a single file through multiple revisions.

Let's say we have a big_file.csv whose content look like this:

a, b, c
1, 2, 3

We first track it using this script:

from lazydata import track

# store the file when loading  
import pandas as pd
df = pd.read_csv(track("big_file.csv"))

print("Data shape:" + str(df.shape))

Change the file content multiple times, for ex:

a, b, c
1, 2, 3
4, 5, 6

And keep executing the script between the multiple revisions:

(dev3.5)  ~/test_lazydata > python my_script.py 
LAZYDATA: Tracking new file `big_file.csv`
Data shape:(1, 3)
(dev3.5)  ~/test_lazydata > vim big_file.csv  # changing file
(dev3.5)  ~/test_lazydata > python my_script.py
LAZYDATA: Tracked file `big_file.csv` changed, recording a new version...
Data shape:(2, 3)
(dev3.5)  ~/test_lazydata > vim big_file.csv  # changing file
(dev3.5)  ~/test_lazydata > python my_script.py
LAZYDATA: Tracked file `big_file.csv` changed, recording a new version...
Data shape:(3, 3)
(dev3.5)  ~/test_lazydata > vim big_file.csv  # changing file
(dev3.5)  ~/test_lazydata > python my_script.py
LAZYDATA: Tracked file `big_file.csv` changed, recording a new version...
Data shape:(4, 3)

A simple ls afterwards points to the mistake:

(dev3.5)  ~/test_lazydata > ls -lah
total 20
drwxrwxr-x  2 zakaria zakaria 4096 sept.  5 16:14 .
drwxr-xr-x 56 zakaria zakaria 4096 sept.  5 16:14 ..
-rw-rw-r--  5 zakaria zakaria   44 sept.  5 16:14 big_file.csv
-rw-rw-r--  1 zakaria zakaria  482 sept.  5 16:14 lazydata.yml
-rw-rw-r--  1 zakaria zakaria  158 sept.  5 16:12 my_script.py

Notice the number of hardlinks to big_file.csv. There should only be one. What is happening is that all the revisions point to the same file.

You can also check ~/.lazydata/data directly for the content of the different files. It'a all the same.

Using Python 2.7.6 and the example script you provide, with a file outside of the github repo (file exists and file path is correct, I checked):

from lazydata import track

with open(track("/home/lg390/tmp/data/some_data_file.txt"), "r") as f:
    print(f.read())

I get

-> % python sample_script.py
Traceback (most recent call last):
  File "sample_script.py", line 1, in <module>
    from lazydata import track
  File "/usr/local/lib/python2.7/dist-packages/lazydata/__init__.py", line 1, in <module>
    from .tracker import track
  File "/usr/local/lib/python2.7/dist-packages/lazydata/tracker.py", line 11
    def track(path:str) -> str:
                  ^
SyntaxError: invalid syntax

I have a project https://github.com/rominf/profanityfilter that could benefit from lazydata. I think it would be cool to move badword dictionaries out of repository and track them alongside hunspell dictionaries with lazydata. The problem is that I want these files to be accessible by end users, that means, I don't want them to be stored in AWS. Instead, I would like them downloaded by http link.

Hello!

First of all thank you for your contribution to the community! I’ve just found out about this and it seems to be a nice project that is growing!

You are probably familiar with dvc (https://github.com/iterative/dvc).

I’ve been investigating it in order to include it in my ML pipeline. Can you explain briefly how/if Lazydata differs from dvc? And any advantages and disadvantages? I understand that there may be some functionalities that maybe are not yet implemented purely due to time constraints or similar. I’m more interested in knowing if there are any differences in terms of paradigm.

Ps- if you have a different channel for these kind of questions please let me know.

Thank you very much!

I've made a PR #13 that you've accepted. Unfortunately, I cannot use the library the easy way because you didn't upload the latest changes on PyPI.

I propose to implement #20 first.

If the package has optional features that require their own dependencies you can use extras_require.

I propose to make use of extras_require for all backends that require dependencies to minimize the number of installed packages. For example, I do not use s3, but all 11 packages are installed, 6 of them are needed for s3.

Here's a start of the azure integration. Haven't written tests yet but let me know what you think. Also sorry for some of the style changes, I have an autoformatter on (black). Let me know if you want me to turn that off.

Ref #18

Would it be possible to wrap the classes implementing snakemake.remote.AbstractRemoteObject (snakemake.remote, AbstractRemoteObject) into lazydata.remote.RemoteStorage class?

This would allow to implement the following remote storage providers in one go (https://snakemake.readthedocs.io/en/stable/snakefiles/remote_files.html):

• Amazon Simple Storage Service (AWS S3): snakemake.remote.S3
• Google Cloud Storage (GS): snakemake.remote.GS
• File transfer over SSH (SFTP): snakemake.remote.SFTP
• Read-only web (HTTP[S]): snakemake.remote.HTTP
• File transfer protocol (FTP): snakemake.remote.FTP
• Dropbox: snakemake.remote.dropbox
• XRootD: snakemake.remote.XRootD
• GenBank / NCBI Entrez: snakemake.remote.NCBI
• WebDAV: snakemake.remote.webdav
• GFAL: snakemake.remote.gfal
• GridFTP: snakemake.remote.gridftp
• iRODS: snakemake.remote.iRODS
• EGA: snakemake.remote.EGA

Pyfilesystem2

Another alternative would be to write a wrapper around pyfilesystem2: https://github.com/PyFilesystem/pyfilesystem2. It supports the following filesystems: https://www.pyfilesystem.org/page/index-of-filesystems/

Builtin

• FTPFS File Transfer Protocol.
• ...

Official

Filesystems in the PyFilesystem organisation on GitHub.

• S3FS Amazon S3 Filesystem.
• WebDavFS WebDav Filesystem.

Third Party

• fs.archive Enhanced archive filesystems.
• fs.dropboxfs Dropbox Filesystem.
• fs-gcsfs Google Cloud Storage Filesystem.
• fs.googledrivefs Google Drive Filesystem.
• fs.onedrivefs Microsoft OneDrive Filesystem.
• fs.smbfs A filesystem running over the SMB protocol.
• fs.sshfs A filesystem running over the SSH protocol.
• fs.youtube A filesystem for accessing YouTube Videos and Playlists.
• fs.dnla A filesystem for accessing accessing DLNA Servers

First, thanks for the amazing package. Exactly what I was looking for!

It would be great to also have a command lazydata track <file1> <file2> ..., which would run lazydata.track() on the specified files. That way, the user can use CLI tools outside of python while still easily tracking the produced files.