pandas-gbq is a package providing an interface to the Google BigQuery API from pandas
$ conda install pandas-gbq --channel conda-forge
$ pip install pandas-gbq
$ pip install git+https://github.com/pydata/pandas-gbq.git
See the pandas-gbq documentation for more details.
I've rewritten the current read_gbq() function using google-cloud-python, which handles the naming of structs and arrays out of the box. For more discussion about this, see: https://github.com/pydata/pandas-gbq/issues/23.
~However, because of the fact that google-cloud-python potentially uses different authentication flows and may break existing behavior, I've left the existing read_gbq() function and and named this new function from_gbq(). If in the future we are able to reconcile the authentication flows and/or decide to deprecate flows that are not supported in google-cloud-python, we can rename this to read_gbq().~
UPDATE: As requested in comment by @jreback (https://github.com/pydata/pandas-gbq/pull/25/files/a763cf071813c836b7e00ae40ccf14e93e8fd72b#r110518161), I deleted old read_gbq() and named my new function read_gbq(), deleting all legacy functions and code.
Added in a few lines to requirements file, but I'll leave it to you @jreback to deal with conda dependency issues which you mentioned in Issue 23.
Let know if any questions or if any tests need to be written. You can confirm that it works by running the following:
q = """
select ROW_NUMBER() over () row_num, struct(a,b) col, c, d, c*d c_times_d, e
from
(select * from
(SELECT 1 a, 2 b, null c, 0 d, 100 e)
UNION ALL
(SELECT 5 a, 6 b, 0 c, null d, 200 e)
UNION ALL
(SELECT 8 a, 9 b, 10.0 c, 10 d, 300 e)
)
"""
df = gbq.read_gbq(q, dialect='standard')
df
| row_num | col | c | d | c_times_d | e | |---------|--------------------|------|------|-----------|-----| | 2 | {u'a': 5, u'b': 6} | 0.0 | NaN | NaN | 200 | | 1 | {u'a': 1, u'b': 2} | NaN | 0.0 | NaN | 100 | | 3 | {u'a': 8, u'b': 9} | 10.0 | 10.0 | 100.0 | 300 |
q = """
select array_agg(a) mylist
from
(select "1" a UNION ALL select "2" a)
"""
df = gbq.read_gbq(q, dialect='standard')
df
| mylist | |--------| | [1, 2] |
q = """
select array_agg(struct(a,b)) col, f
from
(select * from
(SELECT 1 a, 2 b, null c, 0 d, 100 e, "hello" f)
UNION ALL
(SELECT 5 a, 6 b, 0 c, null d, 200 e, "ok" f)
UNION ALL
(SELECT 8 a, 9 b, 10.0 c, 10 d, 300 e, "ok" f)
)
group by f
"""
df = gbq.read_gbq(q, dialect='standard')
df
| col | f | |------------------------------------------|-------| | [{u'a': 5, u'b': 6}, {u'a': 8, u'b': 9}] | ok | | [{u'a': 1, u'b': 2}] | hello |
Confirmed that col_order and index_col still work ~(feel free to pull that out into a separate function since there's now redundant code with read_gbq())~, and I removed the type conversion lines which appear to be unnecessary (google-cloud-python and/or pandas appears to do the necessary type conversion automatically, even if there are nulls; can confirm by examining the datatypes in the resulting dataframes).
We're starting to use BigQuery heavily but becoming increasingly 'bottlenecked' with the performance of moving moderate amounts of data from BigQuery to python.
Here's a few stats:
google-cloud-bigquery - i.e. client.query(query)..to_dataframe()pd.read_sql). That's on standard drivers, nothing like turbodbc involved...so using BigQuery with python is at least an order of magnitude slower than traditional DBs.
We've tried exporting tables to CSV on GCS and reading those in, which works fairly well for data processes, though not for exploration.
A few questions - feel free to jump in with partial replies:
pandas-gbq invest resources into getting a format that's query-able in exploratory workflows that can deal with more reasonable datasets? (or at least encourage Google to)Remove the use of oauth2client and use google-auth library, instead.
Rather than check for multiple versions of the libraries, use the setup.py to specify compatible versions. I believe this is safe since Pandas checks for the pandas_gbq package.
Since google-auth does not use the argparse module to override user authentication flow settings, add a parameter to choose between the web and console flow.
Closes https://github.com/pydata/pandas-gbq/issues/37.
type: bugHi, I'm using Heroku to run a python based ETL process where I'm pushing the contents of a Pandas dataframe into Google BQ using to_gbq. However, it's generating a UnicodeEncodeError with the following stack trace, due to some non-latin characters.
Strangely this works fine on my Mac but when I try to run it on Heroku, it's failing. It seems that for some reason, http.client.py is getting an un-encoded string rather than bytes and therefore, it's trying to encode with latin-1, which is the default but obviously would choke on anything non-latin, like Chinese chars.
2018-01-08T04:54:17.307496+00:00 app[run.2251]:
Load is 100.0% Complete044+00:00 app[run.2251]:
2018-01-08T04:54:20.443238+00:00 app[run.2251]: Traceback (most recent call last):
2018-01-08T04:54:20.443267+00:00 app[run.2251]: File "AllCostAndRev.py", line 534, in
There's a bug with the most recent google bigquery library. Hence, this error occurs in pandas-gbq
ImportError: pandas-gbq requires google-cloud-bigquery: cannot import name 'collections_abc'
Current behavior of to_gbq is fail if the schema of the new data is not equivalent to the current schema. However, this means that the load fails if the new data is missing columns that are present in the current schema. For instance, this may occur when the data source I am using to construct the dataframe only provides non-empty values. Rather than determining the current schema of the GBQ table and adding empty columns to my dataframe, I would like to_gbq to load my data if the columns in the dataframe are a subset of the current schema.
schema function out of verify_schema to support both verify_schema and schema_is_subsetschema_is_subset determines whether local_schema is a subset of remote_schemaschema_is_subset rather than verify_schema to determine if the data can be loadedPROJECT_ID etc are retrieved from an environment variable to facilitate local testingtest_gbq through autopep8 added a row after two class namesOne frustrating thing is having to pass the project_id (among other parameters) every time you write a query. For example, personally, I usually use the same project_id, almost always query with standard sql, and usually turn off verbose. I have to pass those three with every read_gbq, typing which adds up.
Potential options include setting an environment variable and reading from these default settings, but sometimes it can be different each time and fiddling with environment variables feels unfriendly. My suggestion would perhaps be to add a class that can wrap read_gbq() and to_gbq() in a client object. You could set the project_id attribute and dialect and whatever else in the client object, then re-use the object every time you want a query with those settings.
A very naive implementation here in this branch: https://github.com/pydata/pandas-gbq/compare/master...jasonqng:client-object-class?expand=1
Usage would be like:
>>> import gbq
>>> client = gbq.Client(project_id='project-name',dialect='standard',verbose=False)
>>> client.read("select 1")
f0_
0 1
>>> client.read("select 2")
f0_
0 2
>>> client.verbose=True
>>> client.read("select 3")
Requesting query... ok.
Job ID: c7d7e4c0-883a-4e14-b35f-61c9fae0c08b
Query running...
Query done.
Processed: 0.0 B Billed: 0.0 B
Standard price: $0.00 USD
Retrieving results...
Got 1 rows.
Total time taken 1.66 s.
Finished at 2018-01-02 14:06:01.
f0_
0 3
Does that seem like a reasonable solution to all this extra typing or is there another preferred way? If so, I can open up a PR with the above branch.
Thanks, my tired fingers thank you all!
@tswast @jreback @parthea @maxim-lian
Version 0.1.4
This query returns a improperly named dict:
q = """
select struct(a,b) col
from
(SELECT 1 a, 2 b)
"""
df = gbq.read_gbq(q, dialect='standard', verbose=False)
Compare with result from Big Query:
An array of items also get turned into a arrays of dicts sometimes. For example:
q = """
select array_agg(a)
from
(select "1" a UNION ALL select "2" a)
"""
gbq.read_gbq(q, dialect='standard', verbose=False, project_id='project')
outputs:
Compare to Big Query:
These issues may or may not be related?
type: bug help wantedWe're printing in addition to logging, when querying from BigQuery. This makes controlling the output much harder, aside from being un-idiomatic.
Printing in white, logging in red:
https://cloud.githubusercontent.com/assets/5635139/23176541/6028b884-f831-11e6-911a-48aa7741a4da.png
type: feature requestBigquery requires scoped credentials when loading application default credentials
Quick test code below, it will return "invalid token" error. When uncomment the create_scoped() statement, the code run correctly without any error.
# use google default application credentials
export GOOGLE_APPLICATION_CREDENTIALS=/PATH/TO/GOOGLE_DEFAULT_CREDENTIALS.json
import httplib2
from googleapiclient.discovery import build
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
#credentials = credentials.create_scoped('https://www.googleapis.com/auth/bigquery')
http = httplib2.Http()
http = credentials.authorize(http)
service = build('bigquery', 'v2', http=http)
jobs = service.jobs()
job_data = {'configuration': {'query': {'query': 'SELECT 1'}}}
jobs.insert(projectId='projectid', body=job_data).execute()
When attempting to grant pandas access to my GBQ project, I am running into an issue where read_gbq is trying to get default credentials, failing / timing out, then printing out a URL to go to to grant the credentials. Since I'm not running this on google cloud platform, I do not expect to be able to get default credentials. In my case, I only want to run the CLI flow (without having oauth call back to my local server).
Here's the code
>>> import pandas_gbq as gbq
>>> gbq.read_gbq('SELECT 1', project_id=<project_id>, auth_local_webserver=False)
Here's what I see when I trigger a SIGINT once the query is invoked:
File "/usr/lib/python3.5/site-packages/pandas_gbq/gbq.py", line 214, in get_credentials
credentials = self.get_application_default_credentials()
File "/usr/lib/python3.5/site-packages/pandas_gbq/gbq.py", line 243, in get_application_default_credentials
credentials, _ = google.auth.default(scopes=[self.scope])
File "/usr/lib/python3.5/site-packages/google/auth/_default.py", line 277, in default
credentials, project_id = checker()
File "/usr/lib/python3.5/site-packages/google/auth/_default.py", line 274, in <lambda>
lambda: _get_gce_credentials(request))
File "/usr/lib/python3.5/site-packages/google/auth/_default.py", line 176, in _get_gce_credentials
if _metadata.ping(request=request):
File "/usr/lib/python3.5/site-packages/google/auth/compute_engine/_metadata.py", line 73, in ping
timeout=timeout)
File "/usr/lib/python3.5/site-packages/google/auth/transport/_http_client.py", line 103, in __call__
method, path, body=body, headers=headers, **kwargs)
File "/usr/lib/python3.5/http/client.py", line 1106, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
self.endheaders(body)
File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
self._send_output(message_body)
File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
self.send(msg)
File "/usr/lib/python3.5/http/client.py", line 877, in send
self.connect()
File "/usr/lib/python3.5/http/client.py", line 849, in connect
(self.host,self.port), self.timeout, self.source_address)
File "/usr/lib/python3.5/socket.py", line 702, in create_connection
sock.connect(sa)
KeyboardInterrupt
I've also tried setting the env variable GOOGLE_APPLICATIONS_CREDENTIALS to empty. I'm using pandas-gbq version at commit 64a19b.
Hello. I faced the same confusion as #579, so have tried to update the docs.
Not only the BQ DATE type but also the TIME, TIMESTAMP, and FLOAT64 types seems to be wrong.
It seems to be due to a breaking change in google-cloud-bigquery v3.
https://cloud.google.com/python/docs/reference/bigquery/latest/upgrading#changes-to-data-types-loading-a-pandas-dataframe
I have confirmed the correct dtypes as:
>>> import pandas
>>> sql1 = """
SELECT
TRUE AS BOOL,
123 AS INT64,
123.456 AS FLOAT64,
TIME '12:30:00.45' AS TIME,
DATE "2023-01-01" AS DATE,
DATETIME "2023-01-01 12:30:00.45" AS DATETIME,
TIMESTAMP "2023-01-01 12:30:00.45" AS TIMESTAMP
"""
>>> pandas.read_gbq(sql).dtypes
BOOL boolean
INT64 Int64
FLOAT64 float64
TIME dbtime
DATE dbdate
DATETIME datetime64[ns]
TIMESTAMP datetime64[ns, UTC]
dtype: object
>>> sql2 = """
SELECT
DATE "2023-01-01" AS DATE,
DATETIME "2023-01-01 12:30:00.45" AS DATETIME,
TIMESTAMP "2023-01-01 12:30:00.45" AS TIMESTAMP,
UNION ALL
SELECT
DATE "2263-04-12" AS DATE,
DATETIME "2263-04-12 12:30:00.45" AS DATETIME,
TIMESTAMP "2263-04-12 12:30:00.45" AS TIMESTAMP
"""
>>> pandas.read_gbq(sql2).dtypes
DATE object
DATETIME object
TIMESTAMP object
dtype: object
Fixes #579 🦕
api: bigquery size: sWIP PR for discussion: aiming to provide the ability to include a redirect URI, client ID, and client secrets to facilitate the migration away from "out of band" OAuth authentication.
@tswast
See also changes in these repos:
Hi,
I am having problems to install this package on a macos with M1 chip.
The error:
Could not find <Python.h>. This could mean the following:
* You're on Ubuntu and haven't run apt-get install python3-dev.
* You're on RHEL/Fedora and haven't run yum install python3-devel or
dnf install python3-devel (make sure you also have redhat-rpm-config
installed)
* You're on Mac OS X and the usual Python framework was somehow corrupted
(check your environment variables or try re-installing?)
* You're on Windows and your Python installation was somehow corrupted
(check your environment variables or try re-installing?)
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure
× Encountered error while trying to install package. ╰─> grpcio
note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.
Thanks!
api: bigqueryHi, after checking out the pandas_gbq.read_gbq call parametrization I see that I can supply configuration={'dry_run': True} to make the query job to be a dry run.
However it will still attempt to find query destination to try download rows, which in this case will be nonexistent. It would be great if the pandas_gbq would be aware of dry_run and just output the query stats to debug log or return some stats data.
e.g. querying something like this:
pandas_gbq.read_gbq("SELECT * FROM 'my_project_id.billing_ds.cloud_pricing_export'", configuration={'dry_run': True})
still results in the exception
Traceback (most recent call last):
File "big_query_utils.py", line 134, in
Saving data to NUMERIC Field failing with conversion from NoneType to Decimal is not supported
...........
df.to_gbq(project_id=self.client.project,
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas-1.5.1-py3.9-macosx-10.9-x86_64.egg/pandas/core/frame.py", line 2168, in to_gbq
gbq.to_gbq(
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas-1.5.1-py3.9-macosx-10.9-x86_64.egg/pandas/io/gbq.py", line 218, in to_gbq
pandas_gbq.to_gbq(
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas_gbq-0.17.9-py3.9.egg/pandas_gbq/gbq.py", line 1198, in to_gbq
connector.load_data(
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas_gbq-0.17.9-py3.9.egg/pandas_gbq/gbq.py", line 591, in load_data
chunks = load.load_chunks(
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas_gbq-0.17.9-py3.9.egg/pandas_gbq/load.py", line 240, in load_chunks
load_parquet(
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas_gbq-0.17.9-py3.9.egg/pandas_gbq/load.py", line 128, in load_parquet
dataframe = cast_dataframe_for_parquet(dataframe, schema)
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas_gbq-0.17.9-py3.9.egg/pandas_gbq/load.py", line 103, in cast_dataframe_for_parquet
cast_column = dataframe[column_name].map(decimal.Decimal)
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas-1.5.1-py3.9-macosx-10.9-x86_64.egg/pandas/core/series.py", line 4539, in map
new_values = self._map_values(arg, na_action=na_action)
File "/Users/xxxx/.local/lib/python3.9/site-packages/pandas-1.5.1-py3.9-macosx-10.9-x86_64.egg/pandas/core/base.py", line 890, in _map_values
new_values = map_f(values, mapper)
File "pandas/_libs/lib.pyx", line 2918, in pandas._libs.lib.map_infer
TypeError: conversion from NoneType to Decimal is not supported
pandas-gbq version: 0.17.9test = pd.read_gbq('select * from `data-production.dwh_core.transaction_code` limit 1',
project_id='data-production',
dialect='standard',
location='asia-southeast2')
Results in:
ImportError: cannot import name 'external_account_authorized_user' from 'google.auth' (/opt/conda/lib/python3.8/site-packages/google/auth/__init__.py)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-2-99714eec0d64> in <module>
----> 1 import pydata_google_auth
/opt/conda/lib/python3.8/site-packages/pydata_google_auth/__init__.py in <module>
----> 1 from .auth import default
2 from .auth import get_user_credentials
3 from .auth import load_user_credentials
4 from .auth import save_user_credentials
5 from .auth import load_service_account_credentials
/opt/conda/lib/python3.8/site-packages/pydata_google_auth/auth.py in <module>
6 import google.auth.exceptions
7 import google.oauth2.credentials
----> 8 from google_auth_oauthlib import flow
9 import oauthlib.oauth2.rfc6749.errors
10 import google.auth.transport.requests
/opt/conda/lib/python3.8/site-packages/google_auth_oauthlib/__init__.py in <module>
19 """
20
---> 21 from .interactive import get_user_credentials
22
23 __all__ = ["get_user_credentials"]
/opt/conda/lib/python3.8/site-packages/google_auth_oauthlib/interactive.py in <module>
25 import socket
26
---> 27 import google_auth_oauthlib.flow
28
29
/opt/conda/lib/python3.8/site-packages/google_auth_oauthlib/flow.py in <module>
67 import google.oauth2.credentials
68
---> 69 import google_auth_oauthlib.helpers
70
71
/opt/conda/lib/python3.8/site-packages/google_auth_oauthlib/helpers.py in <module>
25 import json
26
---> 27 from google.auth import external_account_authorized_user
28 import google.oauth2.credentials
29 import requests_oauthlib
ImportError: cannot import name 'external_account_authorized_user' from 'google.auth' (/opt/conda/lib/python3.8/site-packages/google/auth/__init__.py)
We had to downgrade the google-auth-oauthlib to 0.5.3:
!pip install google-auth-oauthlib==0.5.3
It seems like the most recent change (October 25) broke something: https://pypi.org/project/google-auth-oauthlib/#history
api: bigqueryread_gbq supports extreme DATETIME values such as 0001-01-01 00:00:00 (#444) (d120f8f)to_gbq allows strings for DATE and floats for NUMERIC with api_method="load_parquet" (#423) (2180836)datetime.date(1, 1, 1) in load_gbq (#442) (e13abaf)to_gbq (#455) (891a00c)to_gbq uses Parquet by default, use api_method="load_csv" for old behavior (#413) (9a65383)to_gbq to a table in a project different from
the API client project. Specify the target table ID as
project.dataset.table to use this feature. (#321, #347)to_gbq. (#321)to_gbq when table has policyTags. (#354)client.dataset deprecation warnings. (#312)google-cloud-bigquery==2.4.* due to query
hanging bug. (#343)object dtype for TIME columns. (#328)INT64 and other standard SQL aliases in
pandas_gbq.to_gbq table_schema argument. (#322)https://pypi.org/project/pandas-gbq/0.14.1/
Source code(tar.gz)dtypes argument to read_gbq. Use this argument to override
the default dtype for a particular column in the query results.
For example, this can be used to select nullable integer columns as
the Int64 nullable integer pandas extension type. (#242, #332)df = pandas_gbq.read_gbq(
"SELECT CAST(NULL AS INT64) AS null_integer",
dtypes={"null_integer": "Int64"},
)
google-cloud-bigquery-storage 2.0 and higher. (#329)pandas to 0.20.1. (#331)google-cloud-bigquery package as
dependencies. Exclude incompatible 2.0 version. (#324, #329)AttributeError with BQ Storage API to download empty results. (#299)private_key argument is used. (#301)max_results argument to pandas_gbq.read_gbq(). Use this
argument to limit the number of rows in the results DataFrame. Set
max_results to 0 to ignore query outputs, such as for DML or DDL
queries. (#102)progress_bar_type argument to pandas_gbq.read_gbq(). Use
this argument to display a progress bar when downloading data.
(#182)google-cloud-bigquery to 1.11.1.
(#296)use_bqstorage_api by closing BigQuery Storage API client after use. (#294)table_schema argument is not modified inplace. (:issue:278)STRING, ARRAY, and STRUCT columns when
there are zero rows. (#285)user-agent with pandas version information. (#281)pytest.raises usage for latest pytest. Fix warnings in tests.
(#282 )read_gbq. ( #269 )google-cloud-bigquery to 1.9.0. ( #247 )pandas to 0.19.0. ( #262 )reauth=True in order to update your credentials to the most recent version. This is required to use new functionality such as the BigQuery Storage API. ( #267 )to_dataframe() from google-cloud-bigquery in the read_gbq() function. ( #247 )to_gbq to contain only a subset of columns, with the rest being populated using the DataFrame dtypes ( #218 ) (contributed by @johnpaton)project_id in to_gbq from provided credentials if available (contributed by @daureg)read_gbq uses the timezone-aware DatetimeTZDtype(unit='ns', tz='UTC') dtype for BigQuery TIMESTAMP columns. ( #269 )use_bqstorage_api to read_gbq. The BigQuery Storage API can be used to download large query results (>125 MB) more quickly. If the BQ Storage API can't be used, the BigQuery API is used instead. ( #133, #270 )private_key parameter is used. #240pydata-google-auth package for authentication. #241pandas_gbq.read_gbq and pandas_gbq.to_gbq in favor of new credentials argument. Instead, create a credentials object using google.oauth2.service_account.Credentials.from_service_account_info or google.oauth2.service_account.Credentials.from_service_account_file. See the authentication how-to guide for examples. (#161, #231 )pandas_gbq.context.dialect to allow overriding the default SQL syntax dialect. (#195, #235)int columns which contain NULL are now cast to float, rather than object type. (#174)DATE, DATETIME and TIMESTAMP columns are now parsed as pandas' timestamp objects (#224)pandas_gbq.Context to cache credentials in-memory, across calls to read_gbq and to_gbq. (#198, #208)DEBUG level. (#204) With BigQuery's release of clustering querying smaller samples of data is now faster and cheaper.True. (#212) This fixes a bug where pandas-gbq could not refresh credentials if the cached credentials were invalid, revoked, or expired, even when reauth=True.read_gbq performance and memory consumption by delegating DataFrame construction to the Pandas library, radically reducing the number of loops that execute in python (#128)read_gbq, particularly for short queries. (#201)SELECT 1 query when running to_gbq. (#202)dialect is not passed in to read_gbq. The default dialect
will be changing from 'legacy' to 'standard' in a future version.
(#195 )to_gbq. This prevents numerical overflow in certain
edge cases. (#192)PyPI release, Conda Forge release
read_gbq when building a dataframe with integer columns on Windows. Explicitly use 64bit integers when converting from BQ types. (#119)read_gbq when querying for an array of floats (#123)read_gbq with configuration argument. Updates read_gbq to account for breaking change in the way google-cloud-python version 0.32.0+ handles query configuration API representation. (#152)to_gbq where seconds were discarded in timestamp columns. (#148)to_gbq when supplying a user-defined schema (#150)verbose parameter in read_gbq and to_gbq. Messages use the logging module instead of printing progress directly to standard output. (#12)Blaze translates a subset of modified NumPy and Pandas-like syntax to databases and other computing systems. Blaze allows Python users a familiar inte
pandas-gbq pandas-gbq is a package providing an interface to the Google BigQuery API from pandas Installation Install latest release version via conda
Arctic TimeSeries and Tick store Arctic is a high performance datastore for numeric data. It supports Pandas, numpy arrays and pickled objects out-of-
Pandas Flavor The easy way to write your own flavor of Pandas Pandas 0.23 added a (simple) API for registering accessors with Pandas objects. Pandas-f
XPandas (extended Pandas) implements 1D and 2D data containers for storing type-heterogeneous tabular data of any type, and encapsulates feature extra
What is Vaex? Vaex is a high performance Python library for lazy Out-of-Core DataFrames (similar to Pandas), to visualize and explore big tabular data
pandas-log The goal of pandas-log is to provide feedback about basic pandas operations. It provides simple wrapper functions for the most common funct
cuDF - GPU DataFrames NOTE: For the latest stable README.md ensure you are on the main branch. Built based on the Apache Arrow columnar memory format,
pandas API on Apache Spark Explore Koalas docs » Live notebook · Issues · Mailing list Help Thirsty Koalas Devastated by Recent Fires The Koalas proje
datatable This is a Python package for manipulating 2-dimensional tabular data structures (aka data frames). It is close in spirit to pandas or SFrame
Scale your pandas workflows by changing one line of code To use Modin, replace the pandas import: # import pandas as pd import modin.pandas as pd Inst
pysparkling Pysparkling provides a faster, more responsive way to develop programs for PySpark. It enables code intended for Spark applications to exe
swifter A package which efficiently applies any function to a pandas dataframe or series in the fastest available manner. Blog posts Release 1.0.0 Fir
pandasql pandasql allows you to query pandas DataFrames using SQL syntax. It works similarly to sqldf in R. pandasql seeks to provide a more familiar
Pandas Profiling Documentation | Slack | Stack Overflow Generates profile reports from a pandas DataFrame. The pandas df.describe() function is great
3.1k Jan 01, 2023
2.9k Dec 23, 2022
260 Jan 01, 2023
25 Mar 14, 2022
7.7k Jan 01, 2023
206 Dec 13, 2022
5.2k Dec 31, 2022
3.2k Jan 04, 2023
1.6k Jan 05, 2023
8.2k Jan 01, 2023
254 Dec 06, 2022
2.2k Jan 04, 2023
1.2k Jan 09, 2023
10k Jan 01, 2023