Meerkat provides fast and flexible data structures for working with complex machine learning datasets.

When using the meerkat from_pandas, things break if you just ran a filter a do not call reset_index(). You get an ambiguous key error when calling from_pandas . I would add some check with a better error message if a user has a non-sequential index of the dataframe.

Overhaul the internals of the Meerkat DataPanel. The changes seek to enable:

1. Vectorized row-wise operations (e.g. slicing, reduction)
2. Simplified I/O and improved latency
3. Clarified view vs. copy behavior
   • We introduce a new spec detailing when users should expect to get views vs. copies (similar to this resource for NumPy) – I'm working on enforcing this spec throughout the codebase.

The new internals are based primarily off the BlockManager class, a dict-like object meant to replace the dictionary we were storing the DataPanel's columns in before. The BlockManager manages links between a DataPanel's columns and data blocks (AbstractBlock, NumpyBlock) where the data is actually stored. It implements consolidate, which takes columns of similar type in a DataPanel and stores their data together in a block, and apply which applies row-wise operations (e.g. getitem) to the blocks in a vectorized fashion. Other important classes:

• BlockRef objects link a block with the BlockManager. These are critical to the functioning of the BlockManager and are the primary type of object passed between the blocks and the block manager. They consists of two things:
  1. A reference to the block (self.block)
  2. A set of columns in the BlockManager whose data live in the Block
• BlockableMixin - a mixin used with AbstractColumn that holds references to a column's block and the columns index in the block
• BlockView - a simple DataClass holding a block and an index into the block. It is typical for new columns to be created from BlockView

Note: I marked this is a WIP because there are still a few more things to be done on this front.

1. Make concat BlockManager aware

Other major changes:

• Removed visible_rows from AbstractColumn,
• Removed _cloneable_kwargs in favor of a unified _clone, _copy, and _view module (cloneable.py)

Issue

It is very natural for users (and developers) to construct new DataPanel objects from existing ones via DataPanel(dp).

Important Aside

An unexpected consequence of this issue is finding a good way to stratify which attributes should be recomputed and which should simply be shallow copied over.

As an example, two attributes that every DataPanel has is _data and _identifier. _data is typically large and heavy-weight, so we will almost always want to shallow copy it. _identifier is quite lightweight and may be unique to different DataPanels, so maybe this is a property we recompute each time in __init__. Note this is just an example, we may want the identifier to persist.

This is especially relevant for subclassing DataPanel. As of PR #57, self.from_batch() is used to construct new DataPanel containers from existing ones with shared underlying data. However, as the PR mentions, self.from_batch() is called by many other ops (_get, merge, concat, etc.), and none of these methods have a seamless way of passing arguments other than data to __init__.

An example of this is EntityDataPanel, where the index_column should be passed from the current instance to the newly constructed instance. Because there is no way to plumb that information through different calls, the initializer of EntityDataPanel gets called with EntityDataPanel(index_column=None) even if the current instance has an index column. This results in a new column "_ent_index" being added to the new EntityDataPanel.

Proposed Solution 1

Implement a private instance method called _clone(data=None, visible_columns=None...) -> DataPanel/subclass which implements the default functionality for how to construct a new DataPanel with the relevant arguments to plumb from current instance to new instance. We can then call self._clone(data=data, visible_columns-optional) instead of self.from_batch() in ops like _get, merge, concat, etc.

Let's consider the EntityDataPanel case. We want to plumb self.index_column from a current EntityDataPanel to all EntityDataPanels constructed in its image. ._clone will look something like

class EntityDataPanel:
    def _clone(self, data=None) -> EntityDataPanel:
        if data is None:
            data = self.data
        return EntityDataPanel(data, identifier=identifier, index_column=self.index_column)

We can then have ops like DataPanel._get() for example use self._clone() instead of self.from_batch(). For example

class DataPanel:
    def _get(self, idx, materialize=False):
        ...
        # example cases where `index` returns a datapanel
        elif isinstance(index, slice):
            # slice index => multiple row selection (DataPanel)
            # return self.from_batch(
            #    {
            #        k: self._data[k]._get(index, materialize=materialize)
            #        for k in self.visible_columns
            #    })
            return self._clone({
                k: self._data[k]._get(index, materialize=materialize)
                for k in self.visible_columns
            })
        ...

Proposed Solution 2

Instead of having developers reimplement ._clone(), we can have them implement something like _state_keys() but for init args. Something like ._clone_kwargs():

class EntityDataPanel:
    def _clone_kwargs(self) -> EntityDataPanel:
        default_kwargs = super()._clone_kwargs()
        default_kwargs.update({"index_column": self.index_column})
        return default_kwargs

class DataPanel:
    def _default_kwargs(self):
        return {"data": self.data, "identifier": self.identifier}

    def _clone(self, **kwargs):
        default_kwargs = self._clone_kwargs()
        if kwargs:
            default_kwargs.update(kwargs)
        return self.__class__(**default_kwargs)

Bug Description When indexing to get a subset of rows from a DataPanel with a complex custom column type, the type of that column is being changed to a ListColumn in the new subset DataPanel

To Reproduce May be difficult to reproduce as it's only occurring for one custom column type that we have.

1. Create complex custom column type (ours is a column where each cell is a time series with categorical values and subclasses mk.CellColumn)
2. Create a DataPanel instance (dp) that has the above column and some data inside of it
3. Index into the DataPanel (dp_subset = dp[0:1])
4. The column type for that specific column in dp_subset has changed to a ListColumn

System Information

• OS: MacOS

@krandiash enable this code to run without errors:

import meerkat as mk import spacy

nlp = spacy.load("en_core_web_sm") doc1 = nlp("Apple is looking at buying U.K. startup for $1 billion") doc2 = nlp("Hello there")

dp = mk.DataPanel({ # 'text': ['The quick brown fox.', 'Jumped over.'], # 'spacy': mk.SpacyColumn([doc1, doc2]), 'list': [{}, {}] })

dp.write('meerkat.dataset') dp2 = dp.read('meerkat.dataset', nlp=nlp)

Add a sort function that can be used to sort the DataPanel by values in a column.

dp = mk.DataPanel({'a': [1, 3, 2], 'b': ['a', 'c', 'b']})
dp.sort('a') # sorted view into the dp

DataPanels no longer rely on visible_columns to create views. This PR removes visible_columns entirely.

Other changes:

• Improve code coverage
  • Reactivate provenance tests
  • DataPanel batch tests
  • Concat tests
  • Merge tests
• Remove Identifiers, Splits and Info from DataPanel and AbstractColumn

Appending to a DataPanel along columns does not work without suffix argument even when the column names do not overlap.

dp = ms.DataPanel({
    'text': ['The quick brown fox.', 'Jumped over.', 'The lazy dog.'],
    'label': [0, 1, 0]
})
dp2 = ms.DataPanel({
    'string': ['The quick brown fox.', 'Jumped over.', 'The lazy dog.'],
    'target': [0, 1, 0]
})
dp.append(dp2, axis=1)

This code throws ValueError. It works when I provide any suffix, although they are not used.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-18-5f32282aa054> in <module>()
----> 1 dp.append(dp2, axis=1)

1 frames
/usr/local/lib/python3.7/dist-packages/mosaic/datapanel.py in append(self, dp, axis, suffixes, overwrite)
    422             if not overwrite and shared:
    423                 if suffixes is None:
--> 424                     raise ValueError()
    425                 left_suf, right_suf = suffixes
    426                 data = {

ValueError:

Adds entity data panel in pipelines folder. Core ideas

• Data panel that has zero or more embedding columns
• Data panel has index panel for functions like iget and icontains for the unique entity id
• Supports appending, from_datapanel, and other data panel methods
• Supports embedding based functions (e.g., cosine nearest neighbors) that returns the metadata.

This PR has two parts:

1. Implementing the DataFrame.rename function. This is an out of place operation that accepts a Dict or Callable mapper argument similar to Pandas. We only support renaming columns since renaming indexes is not applicable for meerkat.

2. Fixing some issues with the Colab notebook. These mostly had to do with the way imagenette was being downloaded. There are still some minor issues here:

   • The imagenette.csv file has image paths such as train/n02979186/n02979186_9036.JPEG instead of imagenette2-160/train/n02979186/n02979186_9036.JPEG. The solution should probably be to go through the .csv file and add "imagenette2-160" before every line (I believe the file paths are coming straight from imagenette download?).
   • Should the files in the downloads folder be deleted after extracted?

Describe the bug Downloading imagenet like this does not work:

dp = mk.datasets.get(
    "imagenet", 
    dataset_dir="/home/ec2-user/data/imagenet1k",
    download=True,
)

Fails with FileNotFoundError: [Errno 2] No such file or directory: '/home/ec2-user/data/imagenet1k/ILSVRC/ImageSets/CLS-LOC/train_cls.txt'

To Reproduce Steps and code snippet that reproduce the behavior:

1. Code snippet '....'
2. Instructions (Run '...')
3. Errors and traceback '....'

See above

Include any relevant screenshots.

Expected behavior Should download imagenet1k to the specified dataset_dir.

System Information

• OS: Linux
• Versions for RG and relevant dependencies meerkat-ml (latest)

Additional context Add any other context about the problem here.

Describe the bug Cannot run Quick start successfully, Import prudces Error: ModuleNotFoundError: No module named 'meerkat.contrib'

To Reproduce

import meerkat as mk
from meerkat.contrib.imagenette import download_imagenette

Expected behavior Quickstart runs succesfully

System Information

• OS: Ubuntu 18.04.5 LTS
• meerkat-ml (v0.2.5)

A call to copy.deepcopy on a datapanel corrupts _block_index of the columns:

dp = mk.DataPanel({
    "a": pd.Series([0,1,2,3]),
    "b": pd.Series([0,1,2,3]),
    "c": pd.Series([0,1,2,3]),
})
dp.consolidate()
print(dp["a"]._block_index)

import copy

dp = copy.deepcopy(dp)
print(dp["a"]._block_index)

There are some examples in audioset who's start time and end time are outside of the length of the video. For example,

balanced_train_segments/YTID=kKf9OprN9nw_st=400.0_et=410.wav ```

When creating the Audioset DataPanel we should check for this and remove those rows. 

I’m envisioning is something in between a map and a LambdaColumn where the computation happens lazily but is cached once it’s computed. Right now, it’s either you do it all up front or you don’t get caching.

This idea was raised @ANarayan who pointed out that it would be helpful for caching feature preprocessing in NLP pipelines.