The code here is pretty similar to #2517, but I wanted to redo some things and it seemed easiest to start with a new pull request. Also the conversation on the other one was getting pretty long.

Main changes from #2517:

• In the previous version, an expression like size_var + 3 created an Expression, but now it creates a Parameter. That is the only major change.
• As briefly discussed in the other PR, selection_point etc now create the entire Parameter.

Main changes overall (from the master branch):

• Include a vegalite/v5 folder with schema 5.2.0. The first commit is just a duplication of the vegalite/v4 folder. I hope that makes it easier to tell what changes were made.
• Move height/width/view from inner charts to the parent chart for LayerChart.
• Introduce parameters. Here are some examples (please ignore the course notes!).
• Moved most code out of Expression and into a new class called OperatorMixin. The goal is to use the same code for both something like expr + 3 (which should produce an Expression) and something like size_var + 3 (which should produce a Parameter). I'm really not sure if the way I accomplished this is natural or not.
• Added a fair amount of ad hoc code to keep the old selection syntax mostly working. I added many warnings.warn(message, DeprecationWarning), but they seem to all be hidden by default, so I have never successfully seen one of these messages displayed. The code would be a little cleaner if we didn't try to keep this old syntax.

If you see something that could be improved, please let me know, because I'm happy to learn about more efficient ways of doing things.

To do:

• When we are happy with the general syntax, I can write some documentation for parameters and add some examples. I also think it would be good to see if some old examples can be simplified. For example, this US population over time I think is more naturally made with a variable parameter as opposed to a selection parameter.
• Decide how forcefully we want to stop users from using the old selection syntax. My warnings.warn approach seems worthless at the moment since the messages don't get displayed by default.
• The only example that is raising an error is scatter plot with minimap. It is failing because it provides an explicit dictionary using the keyword selection, which in the new schema should be param. I think it's not worth the effort of writing code to try to make this example compile, and we should rewrite that example (only two words need to be changed).
• The other tests that I know of which are failing are some render examples (I haven't looked at those at all, nor have I updated Altair Saver) and test_spec_to_vegalite_mimebundle.

Thanks for any comments/requests!

(Draft version only!)

Surprisingly the biggest obstacle so far hasn't been params but a change to layer. I think in the newest Vega-Lite schema, charts in a layer are not allowed to specify height or width, which seems to break many Altair examples. Here is a minimal example that doesn't work:

no_data = pd.DataFrame()

c = alt.Chart(no_data).mark_circle().properties(
    width=100
)

c+c

I don't see a good way to deal with that. Do you have a suggestion?

I've read the list of "breaking changes" for the Vega-Lite 5.0.0 release and don't see anything that seems related to this, so it does make me wonder if maybe I misunderstand the cause of the problem.

Other things:

• I usually try some tests by running things in Jupyter notebook, but since making the change to Vega-Lite 5 that hasn't worked for me. Instead I get the following message in red the first time I try to display a chart, and then subsequent times I just get a blank response: Error loading script: Script error for "vega-util", needed by: vega-lite http://requirejs.org/docs/errors.html#scripterror It does work in Jupyter Lab.
• Some of the code changes have been experiments trying to learn how the old selection fits with the new parameter. It might be best to redo this code later now that I see more of the big picture.

Hi again,

@mattijn and I have tried to update the Pull Request from last week, taking into account your suggestions and trying to get all the tests (that I know about) to work.

Here are some of the main changes from the current Altair release:

• Changed the Vega-Lite schema version to 4.17.0.
• Added definitions of DatumSchemaGenerator and DatumChannelMixin in generate_schema_wrapper.py.
• Updated the loop in generate_vegalite_channel_wrappers here to allow for the possibility of a datum.
• Updated infer_encoding_types from altair/utils/core.py here to recognize Datum class names.
• Changed get_valid_identifier in tools/schemapi/utils.py to deal with some symbols like [] appearing in certain names in the Vega-Lite schema. (Removing those symbols led to some duplicated class names.)
• Updated TopLevelMixin from altair/vegalite/v4/api.py to allow for layer to be repeated, in addition to row and column.
• Updated the Encoding Channel Options part of the docs here. (I wasn't sure if there was a way to auto-generate these groupings of for example Row with Column with Facet.)
• Added some examples to the example gallery here and some additional documentation about mark_arc and layering in repeat here.
• Applied a temporary fix and linked Vega-Lite issue to get test_vegalite_to_vega_mimebundle to work here.

Main outstanding issue that we know of:

• Starting with v4.9, the format of TopLevelRepeatSpec changed in the Vega-Lite schema. We have some code to deal with this by editing the definition of TopLevelRepeatSpec in the downloaded JSON file. We've tried asking on the Vega-Lite Slack channel and as a Vega-Lite GitHub issue, but it seems like the issue is on the Altair side, not the Vega-Lite side. If the rest of the changes seem mostly in good shape, I can focus on finding a more adequate solution.

This is not part of the current Pull Request, but to get some of the tests to work, we made the following changes to altair_viewer:

• Save https://unpkg.com/[email protected]/build/vega-lite.min.js as vega-lite-4.17.0.js in altair_viewer\altair_viewer\scripts
• Add "4.17.0" in listing.json in altair_viewer\scripts

Thank you for any feedback!

New to Altair, but enthusiastic. I'd like to propose the idea of an Altair Hackathon. I don't know the community well enough to organize it. But I imagine a day or weekend spending a large portion of the time developing more examples, and blog posts, along with the usual code base development and issue queue resolution. The API documentation is very concise, and more examples would help grow the user base.

A Google search for:

how to put on a hackathon

yields some promising guides.

Let's get a version 4.2 release candidate together!

In the past, I always just directly created new releases, but it seems Altair has become pretty popular – https://pypistats.org/packages/altair shows nearly 150,000 daily downloads (yikes!) so doing the release candidate route is probably warranted now.

I think we're in pretty good shape right now on the master branch, but I wanted to check to see if there's anything I'm not thinking of. cc/ folks who have been most active recently: @joelostblom @mattijn @ChristopherDavisUCI

Have you had a chance to kick the tires with the VL 4.17 features landed this week? Anything you think we should address before a new release?

This PR addresses the points raised by @joelostblom in #2607 and #2578 (the later can be closed after this PR). In addition, I went through all mark pages to fix any bugs or vega-lite specific parts I could find. I think the pages are now in pretty good shape but it would certainly be nice if someone else can take a thorough look as well. CC @mattijn in case you want to take a look as well.

I have not yet included the sections with the interactive sliders from the vega-lite documentation. Still need to figure out how to best accomplish this. I'll try the suggestions from comment 4 in #2578 and add it to this PR later on or then in a new one.

You can view the updated documentation at https://binste.github.io/altair-docs/ with the exception of the charts in the geoshape section as I did not have geopandas installed. As a sidenote, where would be a good place to add a note that this is now necessary to build the documentation? Maybe we should add it in requirements_dev.txt so it is also included in the docbuild workflow?

Updated version of https://github.com/altair-viz/altair/pull/2671. I believe the functionality is the same, but the code is a little cleaner. @mattijn As always I am happy for any comments!

Here was the description of https://github.com/altair-viz/altair/pull/2671:

This is a draft version (some tests are currently failing) of the strategy suggested by @arvind and @mattijn here for dealing with parameters in multi-view Altair charts. We lift all parameters to the top level. In the case of selection parameters, we give the parameter a "views" property to indicate where the selections should be happening.

Hey,

Thanks for the package, I'm very keen to try it out on my own data. When trying to create a simple histogram with my own data, VegaLite fails on dataframes with more than 5000 rows. Here's a minimal reproducible example:

import altair as alt
import os
import pandas as pd
import numpy as np

lengths = np.random.randint(0,2000,6000)
lengths_list = lengths.tolist()
labels = [str(i) for i in lengths_list]
peak_lengths = pd.DataFrame.from_dict({'coords': labels, 'length': lengths_list},orient='columns')
alt.Chart(peak_lengths).mark_bar().encode(alt.X('lengths:Q', bin=True),y='count(*):Q')

Here's the error:

---------------------------------------------------------------------------
MaxRowsError                              Traceback (most recent call last)
~/anaconda/envs/py3/lib/python3.5/site-packages/altair/vegalite/v2/api.py in to_dict(self, *args, **kwargs)
    259         copy = self.copy()
    260         original_data = getattr(copy, 'data', Undefined)
--> 261         copy._prepare_data()
    262 
    263         # We make use of two context markers:

~/anaconda/envs/py3/lib/python3.5/site-packages/altair/vegalite/v2/api.py in _prepare_data(self)
    251             pass
    252         elif isinstance(self.data, pd.DataFrame):
--> 253             self.data = pipe(self.data, data_transformers.get())
    254         elif isinstance(self.data, six.string_types):
    255             self.data = core.UrlData(self.data)

~/anaconda/envs/py3/lib/python3.5/site-packages/toolz/functoolz.py in pipe(data, *funcs)
    550     """
    551     for func in funcs:
--> 552         data = func(data)
    553     return data
    554 

~/anaconda/envs/py3/lib/python3.5/site-packages/toolz/functoolz.py in __call__(self, *args, **kwargs)
    281     def __call__(self, *args, **kwargs):
    282         try:
--> 283             return self._partial(*args, **kwargs)
    284         except TypeError as exc:
    285             if self._should_curry(args, kwargs, exc):

~/anaconda/envs/py3/lib/python3.5/site-packages/altair/vegalite/data.py in default_data_transformer(data)
    122 @curry
    123 def default_data_transformer(data):
--> 124     return pipe(data, limit_rows, to_values)
    125 
    126 

~/anaconda/envs/py3/lib/python3.5/site-packages/toolz/functoolz.py in pipe(data, *funcs)
    550     """
    551     for func in funcs:
--> 552         data = func(data)
    553     return data
    554 

~/anaconda/envs/py3/lib/python3.5/site-packages/toolz/functoolz.py in __call__(self, *args, **kwargs)
    281     def __call__(self, *args, **kwargs):
    282         try:
--> 283             return self._partial(*args, **kwargs)
    284         except TypeError as exc:
    285             if self._should_curry(args, kwargs, exc):

~/anaconda/envs/py3/lib/python3.5/site-packages/altair/vegalite/data.py in limit_rows(data, max_rows)
     47             return data
     48     if len(values) > max_rows:
---> 49         raise MaxRowsError('The number of rows in your dataset is greater than the max of {}'.format(max_rows))
     50     return data
     51 

MaxRowsError: The number of rows in your dataset is greater than the max of 5000

A quick issues search didn't turn up any hits for MaxRowsError. There is a related issue (#287), but this was a data set with > 300k rows, and I have a measly 35k. Also, the FAQ link referenced in that issue now turns up a 404. For the meantime, does the advice in #249 still apply?

Package info: Running on Altair 2.0.0rc1, JupyterLab 0.31.12-py35_1 conda-forge

In preparation for the Altair 2.0 release, we need some good example visualizations for the documentation! These could be everything from simple one-panel scatter and line plots, to more complicated layered or stacked plots, to more advanced interactive features.

Note that the v2 API is not finalized yet, and so another purpose of creating these is to find bugs in the current package as we prepare for release. If you find anything, please report it on the issues tracker!

I've started a folder for these examples in altair/vegalite/v2/examples/. You can treat simple_scatter.py as a template.

Every example should:

• have a descriptive docstring, which will eventually be extracted for the documentation website.
• define a chart variable with the main chart object (This will be used both in the unit tests to confirm that the example executes properly, and also eventually used to display the visualization on the documentation website).
• not make any external calls to download data within the script (i.e. don't use urllib). You can define your data directly within the example file, generate your data using pandas and numpy.random, or you can use data available in the vega_datasets package.

The easiest way to get started would be to adapt examples from the Vega-Lite example gallery. Or you can feel free to be creative and build your own visualizations.

Note that the new display architecture is still being built; for display troubleshooting please see the wiki page: https://github.com/altair-viz/altair/wiki/Display-Troubleshooting

We'll look forward to your pull requests!

Hi, first of all I really enjoy using altair. I find it really helpful for creating charts of aggregated statistics over different periods of a time-series. However, using it in Jupyter notebooks results in very large files (about 47MB in a notebook rendering only a single chart).

I wonder if the cause of the issue is the size of the data frame I'm using as input - around 67,000 rows. (Note that the aggregation results in a simple bar chart with about 10 bars)

Is there a way to limit the file size of a chart?

Thanks!

Since Layer is the main interface, it would be nice if tab completion on the object only listed relevant pieces of the API so that you can quickly find what plot types are available (e.g. point(), bar(), text(), etc.)

Currently, since it derives from BaseObject the namespace is polluted with all sorts of traitlet stuff that the user probably doesn't care about.

I'd propose something like this:

class LayerObject(BaseObject):
    # traitlet-related stuff goes here
    def __init__(self, *args, **kwargs):
        super(LayerObject, self).__init__(**kwargs)

    # etc.

class Layer(object):
    # non-traitlet-related Layer methods here
    def __init__(self, *args, **kwargs):
        if len(args)==1:
            self.data = args[0]
        self._layerobject = LayerObject(**kwargs)

    def point(self):
        self.mark = 'point'
        return self

    # etc.

The only problem would be if we ever want to pass Layer to some other class this would complicate things. What do you think?

Steps to reproduce:

virtualenv venv &&
cd venv &&
bin/pip install altair rfc3986-validator &&
bin/python <<EOF
import altair as alt
print(alt.Chart().mark_line().properties(usermeta={}).to_json())
EOF

With altair-4.2.0, the above fails with:

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "…/venv/lib/python3.10/site-packages/altair/vegalite/v4/api.py", line 588, in properties
    self.validate_property(key, val)
  File "…/venv/lib/python3.10/site-packages/altair/utils/schemapi.py", line 464, in validate_property
    return jsonschema.validate(value, props.get(name, {}), resolver=resolver)
  File "…/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 1117, in validate
    cls.check_schema(schema)
  File "…/venv/lib/python3.10/site-packages/jsonschema/validators.py", line 231, in check_schema
    raise exceptions.SchemaError.create_from(error)
jsonschema.exceptions.SchemaError: '#/definitions/Dict<unknown>' is not a 'uri-reference'

Failed validating 'format' in metaschema['allOf'][0]['properties']['$ref']:
    {'format': 'uri-reference', 'type': 'string'}

On schema['$ref']:
    '#/definitions/Dict<unknown>'

But here's the twist: if you remove rfc3986-validator from the pip install command in the above repro, it works!

As you can imagine this is a bit of an head-scratcher. Here's what's going on:

jsonschema fails to validate the Vega-lite schema. To be clear, the problem is not the vega-lite output - it's the schema itself that's invalid, because it contains $ref values that are not valid uri-references. In the example above, the $ref is #/definitions/Dict<unknown> which is invalid because the < and > characters are not allowed in a RFC 3986 URI reference.

Even though the schema is invalid, that goes unnoticed most of the time because jsonschema only validates uri-references if the rfc3986-validator or rfc3987 package is installed! This explains why Altair seems to work fine if these packages are missing.

You may wonder how I ended up in this situation. Well this is where things gets a bit worrying, because I ended up triggering this latent bug simply by installing jupyter! This is because, a few months ago, jupyter_events (which jupyter depends on) added a dependency on jsonschema[format-nongpl] (see jupyter/[email protected]), which in turns pulls rfc3986-validator (figuring out that dependency chain was surprisingly hard - see pypa/pip#11683). Hilarity ensues, with the somewhat mind-blowing outcome that Altair is broken when a recent jupyter is also installed.

One workaround is to uninstall rfc3986-validator right after installing the package that pulled it (e.g. jupyter):

pip uninstall rfc3986-validator

This builds on the discussion which started in #2774.

Currently, building and publishing a new Altair version is a manual process as outlined in RELEASING.md. It involves making sure that various version numbers are properly updated, that you have all dependencies installed locally, building and uploading source and wheel distributions, building and uploading the documentation, adjusting the changelog etc. I'd see the following advantages in using GitHub Action workflows for building and uploading the source and wheel distributions as welll as the documentation:

• More reliable build process as it happens in a fresh environment ensuring that the relevant dependencies are installed
• More secure as a local build environment might contain malware
• Credentials can be shared among maintainers, reducing single-person dependency risk
• Automation reduces risk of errors in general and speeds up the process. This would be helpful if we want to have more frequent releases, e.g. for inevitable bug fixes after the first 5.0 release

After implementing it in the Altair repo, I'd suggest to also use similar workflows for the altair companion packages (altair_saver, altair_viewer, etc.) for the same reasons.

Proposal for discussion

To publish a new release, one would still update the version numbers as described in RELEASING.md and add a git tag, e.g. "v5.0.0". As soon as this is pushed to main, the following two new workflows would be executed. So the trigger is "any commit with a git tag starting with "v". As an alternative, these workflows could also be kicked off manually.

• build_and_publish_package.yml: Builds the source and wheel distributions for Altair and then uploads them to Pypi. Can follow this pypi tutorial. Also see wheels.yml in the Pandas repo. The build for conda-forge is triggered automatically by the conda-forge bot as is already the case now in the feedstock repo
• build_and_publish_docs.yml: Builds the documentation and then publishes it by updating the docs repo. A combination of what is in RELEASING.md and sync_website.sh. Also see docbuild-and-upload.yml in the pandas repo.

The relevant credentials could be stored as GitHub Secrets and would be only available to Altair maintainers.

The above process can of course be further automated such as automatically updating version numbers - some packages have much fancier workflows - but maybe it's good to start out simple.

What do you think? Is this helpful? I'm happy to take this on and start a PR but I first wanted to get some input. Also, it requires that the credentials for Pypi are available and access credentials for the docs repo.

Currently all tests are within the Altair package. So once you install the package Altair you also will get all the test files since the test-suite is inclusive.

It is good practice to store your tests outside the application code as is mentioned here: https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code

Stack created with [Sapling]

• -> #2783

Improve MaxRowsError

Summary:

I've modified the error message to give a link to the docs and the command listed there. Most of the time I run into this, I can safely disable it, but its a little inconvenient to search for the command

I updated the README.md. Furthermore, I really like READMEs which are concise, similar to https://github.com/pandas-dev/pandas, i.e. a README which summarizes what the library does, a fully contained example that can be copy-pasted, simple installation instructions, and a reference to the documentation. This is where users can then go to learn more. Afterwards, some additional content for developers. In my opinion, all the rest should be in the documentation so it's easier to find.

Reasoning for the changes is usually in the commit messages but let me know if anything is unclear!

Included content from #1122 in CONTRIBUTING.md

Is the Google Altair Group still something that should be actively recommended for asking questions or only StackOverflow? I think ideally there is one preferred place to ask questions (StackOverflow) and one place to report feature requests and bugs (GH issues). But as the Google Group already has quite some content, maybe we want to keep the reference to it and I could also add it in the Getting Help page in the documentation?