Flask JSONDash
Easily configurable, chart dashboards from any arbitrary API endpoint. JSON config only. Ready to go.
This project is a flask blueprint that allows you to create sleek dashboards without writing any front end (or backend) code. Everything is powered through simple JSON configurations for declaring arbitrary charts.
Features
- Leveraging popular libraries like C3.js and D3.js, and MUCH MORE.
- Also supports templates and iframes
- Only a basic intuitive configuration is required.
- The dashboard layout and blueprint styles are pre-packaged, and provide only the essentials, while getting out of the way.
- Drag-and-drop your layout easily and intuitively
- Multiple layout modes - bootstrap grid based or totally freeform
- Data utilities for munging and manipulating data suitable for various charts
It uses any specified json endpoint to get data from, so long as the payload format is correct.
JSON configurations intro
The configuration JSON provides core functionality and is at the heart of the project. There are several comprehensive examples available in the examples/config directory to give you an idea of how it works, as well as the core configuration documentation. An simple example:
{
"modules": [
{
"type": "timeseries",
"name": "name3",
"width": 510,
"height": 400,
"dataSource": "http://localhost:5001/test1",
"order": 0
}
]
}
(4.0 and later) You can even provide custom inputs to allow interactivity on each chart!
E.g.
{
"modules": [
{
"name": "line",
"height": "400",
"width": "500",
"dataSource": "http://127.0.0.1:5004/custom-inputs",
"override": false,
"guid": "a6eb10e7-26fa-7814-818a-3699b24415c5",
"type": "line",
"inputs": {
"btn_classes": ["btn", "btn-info", "btn-sm"],
"submit_text": "Submit",
"options": [
{
"type": "number",
"name": "entries",
"input_classes": ["form-control", "input-sm"],
"label": "Number of points",
"help_text": "Change the number of points per entry shown"
}
]
}
}
]
}
Which will map to query parameters (entries=10 in this example) that you can use to filter or change what your endpoint returns!
Also note that the order of the inputs in the json will determine their order in html.
Below is an example output using a custom input configuration:
See the examples/config directory for all the supported options.
Demo
If you want to see all/most charts in action, you'll need to fire up the endpoints.py flask app (included) alongside your main app that uses the blueprint, create a new dashboard, then choose the edit raw json option, specifying one of the json files found in examples/config. (This has been tested using mongodb).
Various chart schemas JSON formats
Each chart is very straightforward. Most of the power is leveraged by the various charting libraries that flask-jsondash defers to. See schemas for more detail on how your endpoint json data should be formatted for a given chart type, as well as how to find docs for each supported library.
Usage
Quickstart
Setting database environment variables.
Make sure the following env vars are set:
CHARTS_DB_HOST- The DB server hostname (defaults to 'localhost')CHARTS_DB_PORT- The DB server port (defaults to 27017)CHARTS_DB_NAME- The DB database name (defaults to 'charts')CHARTS_DB_TABLEThe DB collection name (defaults to 'views')CHARTS_ACTIVE_DBThe DB backend to use - options: 'mongo' (default)
Starting DB
Make sure to start so json configuration can be saved.
Starting db: MongoDB
Start however you'd like, but usually mongod will work. Note: you will need to make sure the collection has been created within your mongo instance, and is specified in the CHARTS_DB_TABLE env var, as well as specify your database name under the CHARTS_DB_NAME env var
Download the package and start the app
Method 1 - use provided flask app
git clone https://github.com/christabor/flask_jsondash.git
cd flask_jsondash
virtualenv env
source env/bin/activate
python setup.py install
cd example_app
python app.py
This will setup the app in a virtual environment and run the included test app (app.py) immediately on port 8080.
If you want to import the blueprint into your own existing flask instance:
Method 2 - use your existing app
pip install flask-jsondash
Your app will need to import and register the blueprint, as well as have the appropriate template tags. An example of this can be found here.
Method 3 - Docker
Assuming you have docker and docker-compose installed:
git clone https://github.com/christabor/flask_jsondash.git
make dockerize
This will build the base and services images, setup your docker services and link them together. The endpoints will run on 0.0.0.0:5004 by default, and your app is available at 0.0.0.0:8080.
Note that there are three docker files, a base and then inheriting ones. This is a way to speed up subsequent app-specific builds without having to reinstall python and update apt repos
Note, for any serious usage, you'll always want to configure external volumes for mongodb, so that your data is persisted OUTSIDE of docker.
Python 3.x usage
The above should work, but you'll need to use the python 3.x equivalent for all of the operations; e.g.:
...
virtualenv -p python3 env
python3 setup.py install
python3 app.py
Requirements
Core
- Flask
- Jinja2
Javascript/CSS
These are not included, as you are likely going to have them yourself. If you don't, you'll need to add them:
- Jquery (JS)
- Bootstrap (CSS/JS)
These are necessary and included, based simply on the likelihood they may not already be used:
- JRespond (JS)
- Masonry (JS)
- Jquery UI (CSS/JS)
Starting flask app
Either import and use the blueprint in your own flask app, or run app.py directly to start the app as-is.
Starting the test server
Run endpoints.py if you'd like to test out existing endpoints to link your chart json to.
Using remote AJAX endpoints
See endpoints.py for examples on how to achieve this. If you do not allow CORS on the server-side, all ajax requests will fail.
Customization
Beyond the above outlined configurations that power the core of jsondash, there are more ways to control how the application works.
Flask configuration
Authentication
By default, no authentication is performed for a given action. However, supporting your own custom auth for each type is just a simple config away. Using the flask pattern of injecting configurations into the app.config namespace (in this case, JSONDASH must be specified), you can put whichever functions you want, and only those specified will be checked. Here is a working example:
def can_edit_others(view_id=None):
if view_id == '...' and session.get('user')['name'] in SECRET_ADMINS:
return True
return False
def can_delete_charts():
return session.get('user')['name'] in SECRET_ADMINS
charts_config = dict(
auth=dict(
edit_others=can_edit_others,
delete=can_delete_charts,
),
)
app.config['JSONDASH'] = charts_config
See below for the supported types and their details.
Authentication types
edit_global
This determines if a user can create OR update a chart with the "global" flag set, which will show the dashboard to all users if the appropriate application flags are set (see global config flags below) If no flag is set for allowing global dashboards, then this option will not be available.
delete
Allows deleting of charts.
clone
Allows cloning of charts.
update
Allows updating of charts.
create
Allows creation of new charts.
view
Allows viewing of a chart. The provided function will be passed the id of the view as a view_id kwarg.
edit_others
Allow editing of other creators' charts. The provided function will be passed the id of the view as a view_id kwarg. If the created_by matches the logged in user, it will automatically be allowed, regardless of the auth override.
Metadata configuration
Metadata can be added to the json configuration for further customization purposes. All arbitrary values will expect an accompanying function to be populated with, in the exact same way as the auth functions listed above. They will all be namespaced under the metadata key inside of the app.config['JSONDASH'] dictionary, if specified.
Below is an example of how you can override these fields with your own arbitrary functions. Note: by default, none take arguments. This may change for specific types.
charts_config = dict(
metadata=dict(
created_by=get_username,
),
)
app.config['JSONDASH'] = charts_config
The following metadata overrides are used, but you can also add arbitrary keys and values, which will be saved to the dashboard config, just not necessarily used here.
created_by
This is used to organize views on the front-page by user, if there is such a key present on the configuration. This key is updated and saved if present, null otherwise.
user
This is the current logged in user. This is required for filtering dashboards by user. You must also set the JSONDASH_FILTERUSERS flag to True in app.config.
Global config flags
Below are global app config flags. Their default values are represented in the example working Python code.
app.config['JSONDASH_FILTERUSERS'] = False: for filtering dashboards by the logged in user. See above for setting user data.
app.config['JSONDASH_GLOBALDASH'] = True: for allowing "global" dashboards to be shown. These dashboards must have a created_user of "global" or be overridden (see below).
app.config['JSONDASH_GLOBAL_USER'] = "global": An owner name to use when allowing global dashboards to be seen. This is set on the created_by property in the specific json config. See above for more examples.
app.config['JSONDASH_MAX_PERPAGE'] = 50: The number of results to show per page. Remaining results will be paginated.
Static asset config options
By default, all assets (css/js) will be loaded remotely by popular CDNs recommended for the given charting library.
However, you might want to ensure assets are always available, or cannot access them because of network/proxy issues. If you would like to use your own local assets specified inside of the settings.py file, you can download them, put them in your app somewhere, and then tell jsondash where they should be loaded from (using the standard flask url_for('static', filename=XXX) pattern.)
Just add a static key in your JSONDASH config with these values like so:
app.config['JSONDASH'] = dict(
static=dict(
js_path='js/vendor',
css_path='css/vendor',
)
)
With default flask static settings, this would resolve the url to /static/js/vendor/filename.js for example.
You can use one or the other, but it's recommended to use both or none.
Jinja template configuration
The following blocks are used in the master template:
jsondash_body: required for the entire layout❗ jsondash_css: required for loading the css❗ jsondash_js: required for loading the js❗ jsondash_api_scripts: optional if you want to register callbacks (see below)✔️ jsondash_init: required to initialize the dashboards❗ jsondash_title: optional if you want to override or augment your page title.✔️
You can just check out the example app to see how it all should work.
JavaScript configuration
Custom callbacks
While the point of jsondash is to make front-end coding completely unnecessary, and use serializable declarative configurations for making dashboards, sometimes you need to do one off stuff that requires scripting. A callback module exists to allow this very easily without getting in the way of existing configurations.
You can customize individual charts by adding your own javascript files alongside your existing app that uses this blueprint and then register call backs on a per-chart id basis. All callbacks will be run in the order you register them, after the chart has been loaded and rendered completely.
To get started: override the template block in your template to allow javascript to be executed, and register a callback with your own arguments.
{% block jsondash_api_scripts %}
<script>
jsondash.api.registerCallback('my-chart-guid', function(container, config, myargs){
console.log('Running FIRST callbacK!');
console.log(myargs[0]);
console.log(myargs[1]);
container.style('background-color', 'green');
console.log(config.guid);
}, ['all', 'my', 'optional', 'arguments']);
// Register a second one, which runs after.
jsondash.api.registerCallback('my-chart-guid', function(container, config){
console.log('Running SECOND callbacK!');
});
script>
{% endblock %}
All callbacks will be passed the following arguments order:
container: The d3 selector for the chart container.config: The json object configuration for this chart.args: The array of arguments you supplied when registering the callback.
To see a list of all your callbacks by chart, you can call jsondash.api.listCallbacks();
Custom events
Several events are triggered throughout the process and can be listened to by your own callbacks, or just other code you have embedded in your application:
jsondash.initjsondash.editform.loadedjsondash.widget.addedjsondash.widget.updatedjsondash.widget.deletedjsondash.widget.refreshjsondash.row.addjsondash.row.deletejsondash.preview
See all events in app.js under EVENTS.
Versioning
This project uses semantic versioning for releases. However, the master branch is considered to be unstable as it represents "bleeding edge", with updates, hotfixes, etc... which will eventually get tagged with a release. If you want to use a stable version, make sure to pin the specific release you want to target.
FAQs
Q: "Why'd you choose to expose library X, Y, or Z?"
A: I tried to go for libraries that are pretty widely known and popular. If you are dissatisfied with what's exposed, you can always add your own by embedding any js/css and html in a template, and loading it through the iframe option.
Q: "How do I customize X, Y, Z?"
A: Because of the level of abstraction used here, a lot of charts will naturally be less configurable than if they had been scripted by hand. This is the tradeoff with being able to quickly setup a lot of charts easily.
The goal here is to use intelligent defaults as much as possible, and then allow the most universal aspects to be customized through a common interface.
However, you can inject raw json-friendly configurations if your chart has the override flag set. This will not work for all charts. See configuration options for more.
Keep in mind, many stylistic customizations can be overridden in css, since most all charts are html and/or SVG. And, as mentioned above, you can always use override option, or the iframe/custom option and make your dataSource endpoint return whatever you want, including a full html/js/css pre-rendered template.
Q: "When exposing metadata, why don't you just use the g variable and read from that?"
A: One way this can be done is using the @app.before_request decorator, and populating the g variable with metadata. The problem is that it creates extremely unnecessary overhead.
Tips & tricks
Using the included data utils
Check out data utils for more.
Loading example dashboards config automatically
Run make fixtures.
Using endpoints dynamically
Because the chart builder utilizes simple endpoints, you can use the power of REST to create more complicated views. For example:
curl -XGET http://localhost:8080/api/foo/
could return {"data": [1, 2, 3, 4]}, but you could customize the url by updating the url saved in your dashboard to support query arguments:
curl -XGET http://localhost:8080/api/foo?gt=9
could return {"data": [10, 20, 30, 40]} instead!
Using shared data for performance and simplicity.
Data from a single source can be shared amongst N charts using namespaced "keys" in the payload. See the shared data section and visit an example configuration here for more.
Generating test data
Included are CLI utilities for generating fake charts, etc. You will need to run them like a python package due to their relative import style which is required for py2/p3 compatibility. To run, for example, the model factory generator, run python -m flask_jsondash.model_factories --records 10. For python3.x, just replace that with python3 -m ....
Using the demo mode
If you append the query argument jsondash_demo_mode=1 to your url (e.g. ...?jsondash_demo_mode=1, the UI will automatically hide any dashboard edit buttons and back button. This can be used for example, for displaying on a mounted screen to hide extraneous details.
Embedding charts elsewhere
Similar to jsondash_demo_mode, you can append the query param embeddable (e.g. ?embeddable=1) in your url which will hide all interactive elements, as well as dashboard titles and other ancillary elements.
Note: this will not remove other aspects of your UI that are outside the scope of the jsondash jinja blocks. Those will need to be hidden separately.
Using gist.github.com
While the data is not dynamically generated, you can easily use Github gists (or any raw file from github.com for that matter) to load charts! Check out the kitchensink dashboard to see a real working chart loaded from via gist!
Embedding graphs from other resources
For example, the build tool Jenkins provides a plugin for build statistics. The raw generated png (url is typically of the format https://{JENKINS_SERVER}/view/{VIEW_NAME}/job/{JOB_NAME}/buildTimeGraph/png) can be directly embedded using the iframe chart option.
Other cool stuff
Check out the Makefile for more useful commands.
Performance
Performance metrics are not available, but you can view some "stress test" examples for the example endpoints. The configuration for these are available in examples/config/stresstest.json. Also, the comprehensive examples (plotly, kitchensink) above are very complex dashboards (20-30 charts, webgl, etc), and have been tested in the browser.
A couple observations on stress tests (performed on Macbook Pro / 16gb / 2.7ghz i7):
- Native D3.js handles large datasets very well. It handled 1-2mb json files with no problem.
- Datatables handles extremely large datasets with no problem. Maximum tested before degradation was around 100,000 rows.
- C3.js starts to lag heavily and spends a good 10 seconds and in some cases crashed Google Chrome (with multiple charts on the page) when updwards of 2-300 data points are used per chart (the example config has 10 charts).
Your performance may be better or worse, so just test it out. As always, Your Mileage May Vary.
Debugging/Troubleshooting
My chart won't load even though the url is correct!
If your site is using https (it should be), this is likely caused by an issue with third-parties not using it, but instead running an insecure http web server. This is unfortunately not easy to fix, unless you:
- Make your site insecure by no longer have an SSL certificate (not preferable)
- Tell the owner of the endpoint to enforce SSL on their end and provide https.
To troubleshoot potential javascript parse errors, open up your browser console (In Chrome for example, it's cmd+option+i for Mac, and ctrl+alt+i on Windows) and see if there are any errors. If there are any parse errors, then the format of your json response may be invalid for a given content type. Make sure to check the schemas page for format requirements
My chart is ugly or is flowing outside the container
This is usually only an issue with datatables, particularly when selecting the number of entries to show. The size of the table will grow, and the layout does not account for that, nor should it. The best case here is to determine what size actually makes sense for you and adjust your chart size accordingly.
You can also use the override option supported by this chart, and specify the number of results per page, and the number of entries that can be shown. See the datatables schema docs for more.
Contributing/Development
If you'd like to work on the project, a good place to start is using the example app to develop against. To do this easily, you'll want to setup a virtual environment and setup the package locally, using the develop mode of setuptools. The below should get you started:
git clone github.com/christabor/flask_jsondash.git
cd flask_jsondash
virtualenv env
source env/bin/activate
git checkout -b YOUR_NEW_BRANCH
python setup.py develop
cd example_app
python app.py
And voila! You can now edit the folder directly, and still use it as a normal pip package without having to reinstall every time you change something.
Tests
To run all tests for python 2.7 and 3.x, with coverage, just run tox (assuming tox is installed.)
Python
You can run these tests using pytest (pip install -U pytest) and then in the existing virtualenv, run pytest tests.
If you are having issues with this approach, an alternative would be to install pytest within the projects' virtualenv (assuming you've created one), and then running it like so: python -m pytest tests.
Test coverage
To find coverage information (assuming pytest-cov is installed), you can run: pytest tests -s --cov=flask_jsondash.
Javascript
JS tests are run using the node library Jasmine. To install and run it, you'll need nodejs installed, then the package: npm install -g jasmine. You can then cd into the tests_js folder and run the provided python script python runner.py
![[question]How to plot timeseries data using C3js](https://avatars.githubusercontent.com/u/11693595?v=4)
![new: [Chart options] POST or GET request with custom headers](https://avatars.githubusercontent.com/u/6977223?v=4)
3 Jul 02, 2022
15 Dec 10, 2022
1 Feb 08, 2022
15 Nov 22, 2022
46 Sep 18, 2022
83 Nov 26, 2022
209 Sep 19, 2022
14 Jun 02, 2021
1 Nov 17, 2021
7 Oct 22, 2022
1 Jan 17, 2022
5 Sep 02, 2022
23 Jan 06, 2023
1 Jan 31, 2022
9 Jul 22, 2022
93 Dec 28, 2022
1.3k Dec 26, 2022
164 Dec 12, 2022
1k Jan 02, 2023
22 Dec 21, 2022