support chunked writing (not appending) for np.arrays -> storage providers with the following qualities:

• batched/unbatched
• fixed shape (all samples have same shape)

also adds:

• in .circleci/config.yaml run pytest & pytest-benchmark separately (with --benchmark-skip & --benchmark-only flags)

things this branch does not do:

• appending (writing arrays to a key that already had arrays written to it)
• caching
• index map chunking
• compression

Create a tutorial on Colab

Users should be able to load a dataset, train a model, and upload the dataset. Feel free to start from a small example and then make the example comprehensive.

1. Ability to modify datasets on fly. Datasets are no longer immutable and can be modified over time
2. Larger datasets can now be uploaded as we removed some RAM limiting components from the hub
3. Caching is introduced to improve IO performance.
4. Dynamic shaping enables very large images/data support. You can have large images/data stored in hub.

I have the 1024 and 128 scale pngs from the FFHQ dataset. I'd like to upload this as a hub:// dataset so that you can copy it to the activeloop namespace.

Currently I am considering how to structure the dataset, and what splits it should be uploaded as.

Below is the schema I have used so far. It includes all of the metadata from the original dataset including the URLs to the original files, and the pixel_md5 hashes match when looping back over the dataset and recomputing them.

ds = hub.empty("./ffhq-1024", overwrite=True)

with ds:
    ds.create_tensor("metadata/author", htype="text")
    ds.create_tensor("metadata/country", htype="text")
    ds.create_tensor("metadata/date_crawled", htype="text")
    ds.create_tensor("metadata/date_uploaded", htype="text")
    ds.create_tensor("metadata/license", htype="text")
    ds.create_tensor("metadata/license_url", htype="text")
    ds.create_tensor("metadata/photo_title", htype="text")
    ds.create_tensor("metadata/photo_url", htype="text")

    ds.create_tensor("images/image", htype="image", sample_compression="png")
    ds.create_tensor("images/face_landmarks", dtype=np.float32)
    ds.create_tensor("images/file_md5", htype="text")
    ds.create_tensor("images/file_path", htype="text")
    ds.create_tensor("images/file_url", htype="text")
    ds.create_tensor("images/file_size", dtype=np.int32)
    ds.create_tensor("images/pixel_md5", htype="text")

    ds.create_tensor("thumbs/image", htype="image", sample_compression="png")
    ds.create_tensor("thumbs/face_landmarks", dtype=np.float32)
    ds.create_tensor("thumbs/file_md5", htype="text")
    ds.create_tensor("thumbs/file_path", htype="text")
    ds.create_tensor("thumbs/file_url", htype="text")
    ds.create_tensor("thumbs/file_size", dtype=np.int32)
    ds.create_tensor("thumbs/pixel_md5", htype="text")

    ds.create_tensor("wilds/face_landmarks", dtype=np.float32)
    ds.create_tensor("wilds/face_rect", dtype=np.float32)
    ds.create_tensor("wilds/file_md5", htype="text")
    ds.create_tensor("wilds/file_path", htype="text")
    ds.create_tensor("wilds/file_url", htype="text")
    ds.create_tensor("wilds/file_size", dtype=np.int32)
    ds.create_tensor("wilds/pixel_md5", htype="text")
    ds.create_tensor("wilds/pixel_size", dtype=np.int32)

Does this structure abide by Hub best practices?

Would it be a good idea to also upload a "ffhq-128" without the 1024 images, and "ffhq-meta" without the 128 images also?

>>> next(ds.tensorflow().as_numpy_iterator())
{
  'metadata/author': array([b'Jeremy Frumkin'], dtype=object), 
  'metadata/country': array([b''], dtype=object), 
  'metadata/date_crawled': array([b'2018-10-10'], dtype=object), 
  'metadata/date_uploaded': array([b'2007-08-16'], dtype=object), 
  'metadata/license': array([b'Attribution-NonCommercial License'], dtype=object), 
  'metadata/license_url': array([b'https://creativecommons.org/licenses/by-nc/2.0/'], dtype=object), 
  'metadata/photo_title': array([b'DSCF0899.JPG'], dtype=object), 
  'metadata/photo_url': array([b'https://www.flickr.com/photos/frumkin/1133484654/'], dtype=object), 
  
  'images/image': array([[[  0, 133, 147], ..., [132, 157, 164]]], dtype=uint8), 
  'images/face_landmarks': array([[131.62, 453.8 ], ..., [521.04, 715.26]], dtype=float32), 
  'images/file_md5': array([b'ddeaeea6ce59569643715759d537fd1b'], dtype=object), 
  'images/file_path': array([b'images1024x1024/00000/00000.png'], dtype=object), 
  'images/file_size': array([1488194], dtype=int32), 
  'images/file_url': array([b'https://drive.google.com/uc?id=1xJYS4u3p0wMmDtvUE13fOkxFaUGBoH42'], dtype=object), 
  'images/pixel_md5': array([b'47238b44dfb87644460cbdcc4607e289'], dtype=object), 
  
  'thumbs/image': array([[[  0, 130, 146], ..., [134, 157, 163]]], dtype=uint8), 
  'thumbs/face_landmarks': array([[ 16.4525 ,  56.725  ], ..., [ 65.13   ,  89.4075 ]], dtype=float32), 
  'thumbs/file_md5': array([b'bd3e40b2ba20f76b55dc282907b89cd1'], dtype=object), 
  'thumbs/file_path': array([b'thumbnails128x128/00000/00000.png'], dtype=object), 
  'thumbs/file_size': array([29050], dtype=int32), 
  'thumbs/file_url': array([b'https://drive.google.com/uc?id=1fUMlLrNuh5NdcnMsOpSJpKcDfYLG6_7E'], dtype=object), 
  'thumbs/pixel_md5': array([b'38d7e93eb9a796d0e65f8c64de8ba161'], dtype=object), 
  
  'wilds/face_landmarks': array([[ 562.5,  697.5], ..., [1060.5,  996.5]], dtype=float32), 
  'wilds/face_rect': array([ 667.,  410., 1438., 1181.], dtype=float32), 
  'wilds/file_md5': array([b'1dc0287e73e485efb0516a80ce9d42b4'], dtype=object), 
  'wilds/file_path': array([b'in-the-wild-images/00000/00000.png'], dtype=object), 
  'wilds/file_size': array([3991569], dtype=int32), 
  'wilds/file_url': array([b'https://drive.google.com/uc?id=1yT9RlvypPefGnREEbuHLE6zDXEQofw-m'], dtype=object), 
  'wilds/pixel_md5': array([b'86b3470c42e33235d76b979161fb2327'], dtype=object), 
  'wilds/pixel_size': array([2016, 1512], dtype=int32)
}

Getting the 900GB Wilds images, along with the TFRecords that are pre-resized for each intermediate scale is proving to be harder to acquire. But just hosting the 1024 scale images would already be a huge improvement in making the dataset accessible.

🚨🚨 Feature Request

If your feature will improve HUB

To explore the structure of a dataset it is convenient to have nicer and more informative prints of dataset objects and samples

Description of the possible solution

1) show ds

now

> ds
Dataset(path='hub://activeloop/abalone_full_dataset', tensors=['length', 'diameter', 'height', 'weight'])

Something along the lines would work (taken from SQLlite)

> ds.height
path: "hub://activeloop/abalone_full_dataset", samples:  1532596

tensor    htype        dtype    shape       compression
------    ------       ------   ------      -----------
length    image        uint8    256x256x3   jpeg
diameter  image        float32  512x512x3   zstd
height    image        float32  512x512x3   zstd
weight    class_label  int32    32          None

and in jupyter notebook shown as a table similar to pandas

2) show ds.tensor

now

> ds.height
Tensor(key='Length')

at least provide full information about tensor

> ds.height
Tensor(
    key='height', 
    htype='image', 
    dtype='uint8', 
    shape=(256, 256, 3), 
    sample_compression='jpeg'
)

or to make consistent with 1)

> ds.height
tensor    htype    dtype     shape       compression
------    ------   ------    ------      -----------
height    image    float32   512x512x3   zstd

2) show ds[0:5] sample

> ds[0:5]
    length    diameter     height     weight
    ------    --------     ------     ------
0      0.5    [[0.,...,0]] "sent.."      dog   
0      0.5    [[0.,...,0]] "text a"      dog   
0      0.5    [[0.,...,0]] "text b"      dog   

and in jupyter notebook visualize images (and other htypes)

Notes

• [ ] Feel free to provide a better format for printing dataset, tensor and sample classes
• [ ] Feel free to suggest other important classes/objects need to printed properly for exploring the structure

🚨🚨 Feature Request

• [ ] Related to an existing Issue
• [X] A new implementation (Improvement, Extension)

We should benchmark memory usage when fetching from a Hub dataset.

If your feature will improve HUB

In the near term, well-scoped memory benchmarks will assess new features. In the long term, it can be used to compare performance with other libraries such as Zarr and Tile.

Description of the possible solution

We could start with a client-side benchmark reading from a local volume, perhaps with memory-profiler.

🐛🐛 Bug Report

In the current test sequence, 11 tests fail with the error AttributeError: module 'numcodecs' has no attribute 'MsgPack', however, this error does not exist in colab environments

⚗️ Current Behavior

When pytest . is run on a Windows 10 environment, 11 tests fail and 6 of them have the error message as described above.

Expected behavior/code These errors should not be thrown

⚙️ Environment

• Python version(s):
  • Python 3.7.9
• OS: Windows 10

🖼 Additional context/Screenshots (optional)

Add any other context about the problem here. If applicable, add screenshots to help explain.

Describe the dataset

Add MPII Human Pose Dataset dataset to Hub. So this would work.

import hub
ds = hub.load("username/mpii-human-pose-dataset")

Steps

1. Please take a look at the docs on uploading datasets.

2. Uploading script should be added to examples folder

Example

You can find an example of large dataset loading and upload here:

• https://github.com/activeloopai/Hub/blob/master/examples/coco/upload_coco2017.py

🚨🚨 Feature Request

• [x] A new implementation (Improvement, Extension)

Is your feature request related to a problem?

Hub currently uses Mozilla Public License (MPL), which requires the following header (from Exhibit A of the license) to be attached to source.

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at https://mozilla.org/MPL/2.0/.

We need help appending MPL headers on source (where appropriate).

this is an essential part of the chunk engine. this is a contribution that is narrow in scope (not implementing the whole chunk engine). i also added explicit type checking during pytests using the pytest-mypy package.

this contribution converts bytes -> chunks & has tests to represent as many edge cases as possible.

note: this chunk generator is for chunking with respect to the primary axis. it does not support slicing, but i came up with a modification that will support it.

let's merge this into release/2.0 first to get the ball rolling & i will make another PR with the modification to support slicing.

Describe the dataset

Add IMET 2020 FGVC7 dataset to Hub. So this would work.

import hub
ds = hub.load("username/imet-2020-fgvc7")

Steps

1. Please take a look at the docs on uploading datasets.

2. Uploading script should be added to examples folder

Example

You can find an example of large dataset loading and upload here:

• https://github.com/activeloopai/Hub/blob/master/examples/coco/upload_coco2017.py

🚀 🚀 Pull Request

Checklist:

• [ ] My code follows the style guidelines of this project and the Contributing document
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have kept the coverage-rate up
• [ ] I have performed a self-review of my own code and resolved any problems
• [ ] I have checked to ensure there aren't any other open Pull Requests for the same change
• [ ] I have described and made corresponding changes to the relevant documentation
• [ ] New and existing unit tests pass locally with my changes

Changes

🐛🐛 Bug Report

I'm trying to understand an issue that is making the PyTorch data loader from deeplake throw an index error for some samples unexpectedly. When I try to fetch the data directly from the data set, the behaviour is not reproducible.

The error first appeared during model training. I was able to reproduce it with the following code:

def deeplake_transform(sample_in, patch_size: int, num_seg_classes: int):
    seg_indices = sample_in["masks/label"]
    partial_mask = sample_in["masks/mask"].astype("float32")
    full_mask = np.zeros((num_seg_classes, patch_size, patch_size), dtype=np.float32)
    for i, idx in enumerate(seg_indices):
        full_mask[idx] = partial_mask[i]

    return dict(
        inputs=dict(image=T.ToTensor()(sample_in["images"])),
        targets=dict(
            segmentations=full_mask,
            classifications=sample_in["labels"].astype("float32"),
        ),
    )
data_loader = ds.pytorch(
        transform=deeplake_transform,
        decode_method={"images": "numpy"},
        batch_size=1,
        num_workers=1,
        transform_kwargs={"num_seg_classes": 67, "patch_size": 512},
    )
iter_loader = iter(data_loader)


while True:
    try:
        sample = next(iter_loader)
    except Exception as e:
        print(e)
        break

    idx += 1
    if idx == len(ds):
        print("finished")
        break

The following error is thrown without much context.

Caught IndexError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File ".venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 34, in fetch
    data.append(next(self.dataset_iter))
  File "/home/test/.venv/lib/python3.8/site-packages/deeplake/integrations/pytorch/dataset.py", line 472, in __iter__
    for data in stream:
  File ".venv/lib/python3.8/site-packages/deeplake/core/io.py", line 311, in read
    yield from self.stream(block)
  File "/home/test/.venv/lib/python3.8/site-packages/deeplake/core/io.py", line 355, in stream
    data = engine.read_sample_from_chunk(
  File ".venv/lib/python3.8/site-packages/deeplake/core/chunk_engine.py", line 1528, in read_sample_from_chunk
    return chunk.read_sample(
  File ".venv/lib/python3.8/site-packages/deeplake/core/chunk/uncompressed_chunk.py", line 213, in read_sample
    sb, eb = bps[local_index]
  File ".venv/lib/python3.8/site-packages/deeplake/core/meta/encode/base_encoder.py", line 247, in __getitem__
    self._encoded[row_index], row_index, local_sample_index
IndexError: index 7133 is out of bounds for axis 0 with size 7133

But the following code produces no errors and exhausts the iterator.

for sample in ds:
    try: # try to read all the data that is used in the 
        sample["images"].data()['value']
        sample["masks/mask"].data()['value']
        sample["masks/label"].data()['value']
        sample["labels"].data()['value']
    except:
        break
    

I'm looking for help here since it may be related to the chunk_engine behaviour. It could help if the internal exception handler were more explicit about the error.

⚙️ Environment

• Python version(s): 3.8.10
• OS: Ubuntu 18.04
• IDE: VS-Code
• Packages: [torch==1.13.1, deeplake==3.1.7]

🚀 🚀 Pull Request

Checklist:

• [ ] My code follows the style guidelines of this project and the Contributing document
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have kept the coverage-rate up
• [ ] I have performed a self-review of my own code and resolved any problems
• [ ] I have checked to ensure there aren't any other open Pull Requests for the same change
• [ ] I have described and made corresponding changes to the relevant documentation
• [ ] New and existing unit tests pass locally with my changes

Changes

🚀 🚀 Pull Request

Checklist:

• [ ] My code follows the style guidelines of this project and the Contributing document
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have kept the coverage-rate up
• [ ] I have performed a self-review of my own code and resolved any problems
• [ ] I have checked to ensure there aren't any other open Pull Requests for the same change
• [ ] I have described and made corresponding changes to the relevant documentation
• [ ] New and existing unit tests pass locally with my changes

Changes

🚀 🚀 Pull Request

Checklist:

• [ ] My code follows the style guidelines of this project and the Contributing document
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have kept the coverage-rate up
• [ ] I have performed a self-review of my own code and resolved any problems
• [ ] I have checked to ensure there aren't any other open Pull Requests for the same change
• [ ] I have described and made corresponding changes to the relevant documentation
• [ ] New and existing unit tests pass locally with my changes

Changes

🚀 🚀 Pull Request

Checklist:

• [ ] My code follows the style guidelines of this project and the Contributing document
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have kept the coverage-rate up
• [ ] I have performed a self-review of my own code and resolved any problems
• [ ] I have checked to ensure there aren't any other open Pull Requests for the same change
• [ ] I have described and made corresponding changes to the relevant documentation
• [ ] New and existing unit tests pass locally with my changes

Changes

Resolves #1556