closes #587

• [x] performance diffs
• [x] newline robustness (e.g., appending to the end of a file that has an existing newline)
• [x] bypass DataDescriptor.__iter__

I wanted to be able to add timedeltas to interactive symbols so I made it so.

In [5]: ds.quandl.vix.knowledge_date
Out[5]: 
   knowledge_date
0      1990-01-02
1      1990-01-03
2      1990-01-04
3      1990-01-05
4      1990-01-08
5      1990-01-09
6      1990-01-10
7      1990-01-11
8      1990-01-12
9      1990-01-15
...

In [6]: ds.quandl.vix.knowledge_date + datetime.timedelta(days=1)
Out[6]: 
   knowledge_date
0      1990-01-03
1      1990-01-04
2      1990-01-05
3      1990-01-06
4      1990-01-09
5      1990-01-10
6      1990-01-11
7      1990-01-12
8      1990-01-13
9      1990-01-16
...

datashape already supported this I just needed to add a _(r)?add and _(r)?sub to isdatelike

Impala is a SQL-on-HDFS solution with Python support.

Apparently it's fast http://blog.cloudera.com/blog/2014/05/new-sql-choices-in-the-apache-hadoop-ecosystem-why-impala-continues-to-lead/ .

It also supports Numba via impyla http://blog.cloudera.com/blog/2014/04/a-new-python-client-for-impala/.

Maybe we should support it as a backend. One approach would be to make a SQLAlchemy dialect for it and then depend on the SQLAlchemy compute layer.

Closes #1045

I wanted to try to learn how some of the symbolic stuff worked so I figured this was an easy one to tackle.

In [1]: import blaze as bz

In [2]: ds = bz.Data([('%s', 'a'), ('%s', 'b')], fields=['a', 'b'])

In [3]: ds.a + ds.b
Out[3]: 

0  %sa
1  %sb

In [4]: ds.a * 2
Out[4]: 
      a
0  %s%s
1  %s%s

In [5]: ds.a % ds.b
Out[5]: 

0  a
1  b

Let me know if I this needs more tests or the tests are in the wrong place.

closes #382

• [x] py.test enforces it's own ideas about testing structure by throwing an error during the collection phase of test running. Is this a deal-breaker?
• [x] adapt unittest.skip decorators to equivalent py.test

This changes isidentical into an object identity check. When profiling blaze, I found that this function is called a lot. This actually was in the top 5 tottime calls even with a very small percall count because of how often it was called.

This change makes it so that all expression objects are weakly cached by their arguments. This means that any two objects that are identical will have the same object id. This will save memory by reducing copies of the same object and improve performance because is checks are extremely fast.

Many backends support string matching.

• Python uses the regex module re
• Pandas also uses re?
• SQL uses LIKE %text%
• Mongo I'm sure has something

This probably warrants a Blaze operation

Hello,

So I really like this project but it seems to have been quite quiet lately.

I spent some time today un-pinning all the packages and getting the tests to pass again. One caveat is that odo has a problem. For this, I've just odo from a repository I built. Obviously, this is not ideal.

Let me know if this is helpful. This is an amazing set of tools.

Text matching via globstrings

Example

In [1]: from blaze import *

In [2]: L = [[1, 'Alice',   100],
     [2, 'Bob',    -200],
     [3, 'Charlie', 300],
     [4, 'Dennis',  400],
     [5, 'Edith',  -500]]

In [3]: t = Table(L, columns=['id', 'name', 'amount'])

In [4]: t.like(name='Al*')
Out[4]: 
   id   name  amount
0   1  Alice     100

In [5]: t.like(name='*e*')
Out[5]: 
   id     name  amount
0   1    Alice     100
1   3  Charlie     300
2   4   Dennis     400

I updated blaze today (blaze {0.6.6_212_g4b7c730 -> 0.8.0}) and I got a new error.

import blaze as bz
apps = bz.Data("mongodb://localhost:27017/API::apps", dshape = 'var * {name: string}')

That used to run fine but now I get this traceback:

Traceback (most recent call last):

  File "<ipython-input-1-81a88c5ec220>", line 1, in <module>
    runfile('/Users/Dalton/Desktop/errorTest.py', wdir='/Users/Dalton/Desktop')

  File "/Users/Dalton/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 682, in runfile
    execfile(filename, namespace)

  File "/Users/Dalton/anaconda/lib/python2.7/site-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 78, in execfile
    builtins.execfile(filename, *where)

  File "/Users/Dalton/Desktop/errorTest.py", line 10, in <module>
    apps = bz.Data("mongodb://localhost:27017/API::apps", dshape = 'var * {name: string}')

  File "/Users/Dalton/anaconda/lib/python2.7/site-packages/blaze/interactive.py", line 100, in Data
    result = result[field]

  File "/Users/Dalton/anaconda/lib/python2.7/site-packages/blaze/expr/expressions.py", line 108, in __getitem__
    raise ValueError("Not understood %s[%s]" % (self, key))

ValueError: Not understood _1[apps]

Let me know if there's anything I can do to be helpful.

I'm new to blaze, so pardon my ignorance here. I have no idea if I have to report this to odo/datashape or something else.

I'm using blaze.Data on a postgresql table ("postgresql://"). When I try to get some data off the table with list(head(10)); in 50% of the cases (without any change on the db), I get this error:

  File "/usr/local/lib/python2.7/dist-packages/odo/into.py", line 122, in curried_into
    return into(o, other, **merge(kwargs2, kwargs1))
  File "/usr/local/lib/python2.7/dist-packages/multipledispatch/dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/blaze/interactive.py", line 309, in into
    return into(a, result, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/multipledispatch/dispatcher.py", line 164, in __call__
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/odo/into.py", line 25, in into_type
    return convert(a, b, dshape=dshape, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/odo/core.py", line 30, in __call__
    return _transform(self.graph, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/odo/core.py", line 46, in _transform
    x = f(x, excluded_edges=excluded_edges, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/odo/convert.py", line 21, in dataframe_to_numpy
    dtype = dshape_to_numpy(dshape or discover(df))
  File "/usr/local/lib/python2.7/dist-packages/odo/numpy_dtype.py", line 55, in dshape_to_numpy
    for name, typ in zip(ds.names, ds.types)])
  File "/usr/local/lib/python2.7/dist-packages/odo/numpy_dtype.py", line 26, in unit_to_dtype
    return unit_to_dtype(str(ds).replace('int', 'float').replace('?', ''))
  File "/usr/local/lib/python2.7/dist-packages/odo/numpy_dtype.py", line 22, in unit_to_dtype
    ds = dshape(ds)
  File "/usr/local/lib/python2.7/dist-packages/datashape/util.py", line 49, in dshape
    ds = parser.parse(o, type_symbol_table.sym)
  File "/usr/local/lib/python2.7/dist-packages/datashape/parser.py", line 575, in parse
    dsp.raise_error('Invalid datashape')
  File "/usr/local/lib/python2.7/dist-packages/datashape/parser.py", line 57, in raise_error
    self.ds_str, errmsg)
datashape.error.DataShapeSyntaxError: 

  File <nofile>, line 1
    float16
    ^

DataShapeSyntaxError: Invalid datashape

I actually wonder why this error is not reproducible. Looks like odo is randomly choosing a different conversion/coercion route? In fact, it's so random I cannot even determine whether there's a specific column type that could cause the issue.

I just upgraded all my python libraries, and now my previous code is started to fail. I'm using blaze with pandas. Here is my method code

blaze.data(res)

res contains below data

             col1         age  ...                                           col31  year
0            yes    55-64  ...                                             NaN  2011
1             no    25-34  ...                                             NaN  2011
2             no    55-64  ...                                             NaN  2011

I'm using below dependencies

- python=3.6.11=h4d41432_2_cpython
- blaze=0.11.3=py36h4e06776_0
- odo=0.5.1=py36h90ed295_0
- pandas=1.0.5=py36h0573a6f_0
- numpy=1.19.1=py36h3849536_2

I'm getting the following error

  File "cytoolz/functoolz.pyx", line 667, in cytoolz.functoolz.pipe
  File "cytoolz/functoolz.pyx", line 642, in cytoolz.functoolz.c_pipe
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/blaze/interactive.py", line 153, in data
    dshape = discover(data_source)
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/multipledispatch/dispatcher.py", line 278, in __call__
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/odo/backends/pandas.py", line 31, in discover_dataframe
    for k in df.columns])
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/odo/backends/pandas.py", line 31, in <listcomp>
    for k in df.columns])
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/odo/backends/pandas.py", line 23, in dshape_from_pandas
    dshape = datashape.CType.from_numpy_dtype(col.dtype)
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/datashape/coretypes.py", line 779, in from_numpy_dtype
    if np.issubdtype(dt, np.datetime64):
  File "/home/ubuntu/miniconda/envs/my_env/lib/python3.6/site-packages/numpy/core/numerictypes.py", line 388, in issubdtype
    arg1 = dtype(arg1).type
TypeError: Cannot interpret 'CategoricalDtype(categories=['no', 'yes'], ordered=False)' as a data type

It's failing to parse second column values.

find . -iname '*.py'  | xargs -P 4 -I{} python -Wall -m py_compile {} 

./docs/gh-pages.py:124: DeprecationWarning: invalid escape sequence \#
  branch = re.match('\# On branch (.*)$', status).group(1)
./versioneer.py:467: DeprecationWarning: invalid escape sequence \s
  LONG_VERSION_PY['git'] = '''
./blaze/expr/collections.py:102: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if self._key is () or self._key is None:

$ rg "from collections.*(Awaitable|Coroutine|AsyncIterable|AsyncIterator|AsyncGenerator|Hashable|Iterable|Iterator|Generator|Reversible|Sized|Container|Callable|Collection|MutableSet|Mapping|MutableMapping|MappingView|KeysView|ItemsView|ValuesView|Sequence|MutableSequence|ByteString)\b"


blaze/utils.py
4:from collections import Iterator

blaze/compute/pandas.py
24:from collections import defaultdict, Iterable

blaze/compute/chunks.py
5:from collections import Iterator, Iterable

blaze/compute/sql.py
20:from collections import Iterable

blaze/compute/csv.py
8:from collections import Iterator, Iterable

blaze/compute/core.py
3:from collections import defaultdict, Iterator, Mapping

blaze/compute/bcolz.py
26:from collections import Iterator, Iterable

blaze/compute/python.py
14:from collections import Iterable, Mapping
24:from collections import Iterator

blaze/compute/json.py
6:from collections import Iterator

blaze/compute/numpy.py
3:from collections import Iterable

blaze/expr/arrays.py
3:from collections import Iterable

blaze/expr/literal.py
3:from collections import Iterator, Mapping

blaze/expr/core.py
3:from collections import Mapping, OrderedDict

blaze/expr/expressions.py
3:from collections import Mapping

blaze/tests/test_cached.py
4:from collections import Iterator

blaze/compute/tests/test_csv_compute.py
11:from collections import Iterator

blaze/compute/tests/test_python_compute.py
2:from collections import Mapping
11:from collections import Iterator, Iterable

blaze/server/serialization/json_dumps_trusted.py
3:from collections import Callable

Dear All,

I'm new to Anaconda/Python environment and wanted to connect to remote HDFS/Hive server. After googling found that we should have pyhive installed on Anaconda. Tried below execution and stuck with following error

(C:\ProgramData\Anaconda3) C:\WINDOWS\system32>conda install -c blaze pyhive Fetching package metadata .................

PackageNotFoundError: Package missing in current win-64 channels:

• pyhive

Close matches found; did you mean one of these?

pyhive: pyfive, pydrive