re is needed to do regular expressions. Then re has to be imported after the datar.

from datar.all import *
import re

I always import re first.Then id doesn't work after being overwritten. Sometimes I use re in function like this :

def test(x,y) 
   re.sub(..)
   re.replace(..)
   return ...

Sometimes I wanna keep all the duplicated rows. While in pandas, done like this mtcarss[mtcars.duplicated(keep=False)] In datar, it does not work.

from datar.all import * 
from datar.datasets import mtcars

mtcars >> select('cyl','hp','gear','disp')>> filter(f.duplicated(keep=False))

But in the follow two ways,it works.

# 1  f.series 

mtcars >> select('cyl','hp','gear','disp')>> filter(f.cyl.duplicated(keep=False))

# 2 select all the columns 

mtcars >> select('cyl','hp','gear','disp')>> filter(f['cyl'].duplicated(keep=False))

It seems that only series can be passed to the filter

Discussed in https://github.com/pwwang/datar/discussions/136

@register_func(None, context=Context.EVAL)
def weighted_mean(
    x: NumericOrIter,
    w: NumericOrIter = None,
    na_rm: bool = False,
) 

Piping into verbs like mutate seems to be broken in 0.9.0. Passing in the dataframe directly works, but still generates a warning.

>>> import datar.all
>>> import datar.datasets
>>> datar.datasets.mtcars >> datar.all.mutate(x = 3)
/home/bizdev/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pipda/utils.py:68: VerbCallingCheckWarning: Failed to detect AST node calling `mutate`, assuming a normal call.
  warnings.warn(
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/bizdev/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pipda/verb.py", line 216, in __call__
    raise TypeError(f"{self.__name__}() missing at least one argument.")
TypeError: mutate() missing at least one argument.
>>> datar.__version__
'0.9.0'
>>> import pipda
>>> pipda.__version__
'0.7.6'

>>> datar.all.mutate(datar.datasets.mtcars, x = 3)
/home/bizdev/.pyenv/versions/3.9.13/lib/python3.9/site-packages/pipda/utils.py:68: VerbCallingCheckWarning: Failed to detect AST node calling `mutate`, assuming a normal call.
  warnings.warn(
                          mpg     cyl      disp      hp      drat        wt      qsec      vs      am    gear    carb       x
                    <float64> <int64> <float64> <int64> <float64> <float64> <float64> <int64> <int64> <int64> <int64> <int64>
Mazda RX4                21.0       6     160.0     110      3.90     2.620     16.46       0       1       4       4       3
Mazda RX4 Wag            21.0       6     160.0     110      3.90     2.875     17.02       0       1       4       4       3

I am trying to run code on RStudio R notebooks with datar but the code does not run. I would like to use R notebooks to highlight parts of the datar code and run separately.

@pwwang what is the issue with running datar on RStudio.

Hi @pwwang , do you have plans to add lubridate commands to datar?

I am trying to convert the Date column on stock time series data to date time with datar mutate.

Data from yahoo finance

import pandas as pd
from datar.all import *

aapl = pd.read_csv("AAPL.csv")

aapl.Date = pd.to_datetime(aapl.Date.astype('str')) # with pandas this works to change the data type to datetime

aapl = aapl >> mutate(Date = as_datetime(f.Date))  # this does not work and shows error message

aapl = aapl >> mutate(Date = as_date(f.Date))  #this does not work and does not show error message

Issue

I just upgraded datar from 0.4.3 to 0.4.4 with pip install -U datar and got error ImportError: cannot import name 'VarnameException' when importing datar with code from datar.all import *

below is the error message:

ImportError: cannot import name 'VarnameException' from 'varname' (C:\Anaconda3\lib\site-packages\varname\__init__.py)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-268be173473d> in <module>
----> 1 from datar.all import *
      2 from datar.datasets import mtcars
      3 mtcars  >> mutate(mpl = f.mpg/4)

C:\Anaconda3\lib\site-packages\datar\all.py in <module>
      7 from .base import *
      8 from .base import _warn as _
----> 9 from .datar import *
     10 from .dplyr import _no_warn as _
     11 from .dplyr import _builtin_names as _dplyr_builtin_names

C:\Anaconda3\lib\site-packages\datar\datar\__init__.py in <module>
      1 """Specific verbs/funcs from this package"""
      2 
----> 3 from .verbs import get, flatten, drop_index
      4 from .funcs import itemgetter

C:\Anaconda3\lib\site-packages\datar\datar\verbs.py in <module>
      9 from ..core.contexts import Context
     10 from ..core.grouped import DataFrameGroupBy
---> 11 from ..dplyr import select, slice_
     12 
     13 

C:\Anaconda3\lib\site-packages\datar\dplyr\__init__.py in <module>
      4 from .across import across, c_across, if_all, if_any
      5 from .arrange import arrange
----> 6 from .bind import bind_cols, bind_rows
      7 from .context import (
      8     cur_column,

C:\Anaconda3\lib\site-packages\datar\dplyr\bind.py in <module>
     15 from ..core.names import repair_names
     16 from ..core.grouped import DataFrameGroupBy
---> 17 from ..tibble import tibble
     18 
     19 

C:\Anaconda3\lib\site-packages\datar\tibble\__init__.py in <module>
      1 """APIs for R-tibble"""
----> 2 from .tibble import tibble, tibble_row, tribble, zibble
      3 from .verbs import (
      4     enframe,
      5     deframe,

C:\Anaconda3\lib\site-packages\datar\tibble\tibble.py in <module>
      4 
      5 from pandas import DataFrame
----> 6 from varname import argname, varname, VarnameException
      7 
      8 import pipda

ImportError: cannot import name 'VarnameException' from 'varname' (C:\Anaconda3\lib\site-packages\varname\__init__.py)

Expected

Expect datar to work without issue after upgrading to new versions.

when the case_when is used the output is not as expected.

Code to replicate

mtcars >> mutate(gas_milage = case_when(
                            f.mpg >21  and f.mpg <= 22, "ok",
                            f.mpg >22, "best",
                            True, "other"

))

Issue: The last line in the output does not meet the f.mpg >21 and f.mpg <= 22, "ok" but it is still applied the "ok" label

Expected result

Only rows meeting the f.mpg >21 and f.mpg <= 22, "ok" condition labeled "ok", all other rows not meeting any condition should be labeled "other"

Hey @pwwang , does datar have a function to display the translation to pandas commands?

I believe it would be a very useful addition for many reasons, one being that it would help a lot datar and pandas users to work together in a project.

In R, an analogous function would be dplyr::show_query(), which shows the translation to SQL, like in the example below:

df <- dbplyr::lazy_frame(mtcars)
df |> dplyr::select(mpg) |>  dplyr::show_query()
#> <SQL>
#> SELECT `mpg`
#> FROM `df`

Thank you

seems that the pipe operator doesn't work when using datar in virtual anaconda environments. here's a snippet of running the example code ran in anaconda prompt:

from datar.all import f, mutate, filter, if_else, tibble

[2021-08-03 19:57:12][datar][WARNING] Builtin name "filter" has been overriden by datar.

df = tibble(
    x=range(4),
    y=['zero', 'one', 'two', 'three']
)

C:\ProgramData\Anaconda3\envs\conda_start\lib\site-packages\pipda\utils.py:159: UserWarning: Failed to fetch the node calling the function, call it with the original function. warnings.warn(

df >> mutate(z=f.x)

C:\ProgramData\Anaconda3\envs\conda_start\lib\site-packages\pipda\utils.py:159: UserWarning: Failed to fetch the node calling the function, call it with the original function. warnings.warn( Traceback (most recent call last): File "", line 1, in File "C:\ProgramData\Anaconda3\envs\conda_start\lib\site-packages\pipda\register.py", line 396, in wrapper return calling_rule(generic, args, kwargs, envdata) File "C:\ProgramData\Anaconda3\envs\conda_start\lib\site-packages\pipda_calling.py", line 93, in verb_calling_rule3 return generic(*args, **kwargs) File "C:\ProgramData\Anaconda3\envs\conda_start\lib\functools.py", line 872, in wrapper raise TypeError(f'{funcname} requires at least ' TypeError: _not_implemented requires at least 1 positional argument

mutate(df, z=f.x)

>         x        y       z
>   <int64> <object> <int64>
> 0       0     zero       0
> 1       1      one       1
> 2       2      two       2
> 3       3    three       3

pandas 1.2.3 python 3.8.1

ps thanks a lot for the package, hopefully the issue can be closed soon :)

@GitHunter0

I'm just having an issue with multi-line execution of datar code in VScode.

If a run this line by line, it works smoothly.

from datar.all import (f, mutate, tibble, fct_infreq, fct_inorder, pull)
df = tibble(var=['b','b','b','c','a','a'])
df = df >> mutate(fct_var = f['var'].astype("category"))

However, if I select all the lines and execute them, it returns:

C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\utils.py:161: UserWarning: Failed to fetch the node calling the function, call it with the original function.

>>> df = df >> mutate(fct_var = f['var'].astype("category"))

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\register.py", line 396, in wrapper
    return calling_rule(generic, args, kwargs, envdata)
  File "C:\Users\user_name\miniconda3\envs\py38\lib\site-packages\pipda\_calling.py", line 93, in verb_calling_rule3
    return generic(*args, **kwargs)
  File "C:\Users\user_name\miniconda3\envs\py38\lib\functools.py", line 872, in wrapper
    raise TypeError(f'{funcname} requires at least '
TypeError: _not_implemented requires at least 1 positional argument

Originally posted by @GitHunter0 in https://github.com/pwwang/datar/discussions/48#discussioncomment-1286583

datar version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of datar and its backends.

Issue Description

When a column (in this case 'x2') is not present, relocate() with any_of() does not respect the order passed.

import datar.all as d
from datar import f
import pandas as pd

df = pd.DataFrame({'x1': [1], 'x4': [4], 'x3': [3]})

df >> d.relocate(d.any_of(['x1', 'x2', 'x3', 'x4']))
#>        x1      x4      x3
  <int64> <int64> <int64>
0       1       4       3

Expected Behavior

The new order should be 'x1', 'x3', 'x4'

Installed Versions

When I create grouped data with datar's group_by(), I get an undesirable DataFrameGroupBy element instead of a DataFrame. It is not desirable to have a DataFrameGroupBy in VSCode because the dataframe cannot be clicked on the Variables Window of VSCode to see the entire dataframe, whereas the mtcars can be click to exposed the full dataset because it is a DataFrame.

The code below creates grouped data in datar and grouped data in pandas; However, datar creates a DataFrameGroupBy instead of a dataframe.

from datar.all import *
from datar.datasets import mtcars
datar_group = mtcars >> group_by(f.hp) >> count()
pandas_group = mtcars.groupby('hp').size().reset_index().rename(columns = {0:"n"})