Evidently
Interactive reports and JSON profiles to analyze, monitor and debug machine learning models.
Docs | Join Discord | Newsletter | Blog | Twitter
What is it?
Evidently helps analyze machine learning models during validation or production monitoring. The tool generates interactive visual reports and JSON profiles from pandas DataFrame or csv files. Currently 6 reports are available.
1. Data Drift
Detects changes in feature distribution. 
2. Numerical Target Drift
Detects changes in numerical target and feature behavior. 
3. Categorical Target Drift
Detects changes in categorical target and feature behavior. 
4. Regression Model Performance
Analyzes the performance of a regression model and model errors. 
5. Classification Model Performance
Analyzes the performance and errors of a classification model. Works both for binary and multi-class models. 
6. Probabilistic Classification Model Performance
Analyzes the performance of a probabilistic classification model, quality of model calibration, and model errors. Works both for binary and multi-class models. 
Installing from PyPI
MAC OS and Linux
Evidently is available as a PyPI package. To install it using pip package manager, run:
$ pip install evidently
The tool allows building interactive reports both inside a Jupyter notebook and as a separate HTML file. If you only want to generate interactive reports as HTML files or export as JSON profiles, the installation is now complete.
To enable building interactive reports inside a Jupyter notebook, we use jupyter nbextension. If you want to create reports inside a Jupyter notebook, then after installing evidently you should run the two following commands in the terminal from evidently directory.
To install jupyter nbextention, run:
$ jupyter nbextension install --sys-prefix --symlink --overwrite --py evidently
To enable it, run:
jupyter nbextension enable evidently --py --sys-prefix
That's it!
Note: a single run after the installation is enough. No need to repeat the last two commands every time.
Note 2: if you use Jupyter Lab, you may experience difficulties with exploring report inside a Jupyter notebook. However, the report generation in a separate .html file will work correctly.
Windows
Evidently is available as a PyPI package. To install it using pip package manager, run:
$ pip install evidently
The tool allows building interactive reports both inside a Jupyter notebook and as a separate HTML file. Unfortunately, building reports inside a Jupyter notebook is not yet possible for Windows. The reason is Windows requires administrator privileges to create symlink. In later versions we will address this issue.
Getting started
Jupyter Notebook
To start, prepare your data as two pandas DataFrames. The first should include your reference data, the second - current production data. The structure of both datasets should be identical.
- For Data Drift report, include the input features only.
- For Target Drift reports, include the column with Target and/or Prediction.
- For Model Performance reports, include the columns with Target and Prediction.
Calculation results can be available in one of the two formats:
- Option 1: an interactive Dashboard displayed inside the Jupyter notebook or exportable as a HTML report.
- Option 2: a JSON Profile that includes the values of metrics and the results of statistical tests.
Option 1: Dashboard
After installing the tool, import Evidently dashboard and required tabs:
import pandas as pd
from sklearn import datasets
from evidently.dashboard import Dashboard
from evidently.tabs import DataDriftTab
iris = datasets.load_iris()
iris_frame = pd.DataFrame(iris.data, columns = iris.feature_names)
To generate the Data Drift report, run:
iris_data_drift_report = Dashboard(tabs=[DataDriftTab])
iris_data_drift_report.calculate(iris_frame[:100], iris_frame[100:], column_mapping = None)
iris_data_drift_report.save("reports/my_report.html")
To generate the Data Drift and the Categorical Target Drift reports, run:
iris_data_and_target_drift_report = Dashboard(tabs=[DataDriftTab, CatTargetDriftTab])
iris_data_and_target_drift_report.calculate(iris_frame[:100], iris_frame[100:], column_mapping = None)
iris_data_and_target_drift_report.save("reports/my_report_with_2_tabs.html")
If you get a security alert, press "trust html". HTML report does not open automatically. To explore it, you should open it from the destination folder.
To generate the Regression Model Performance report, run:
regression_model_performance = Dashboard(tabs=[RegressionPerfomanceTab])
regression_model_performance.calculate(reference_data, current_data, column_mapping = column_mapping)
You can also generate a Regression Model Performance for a single DataFrame. In this case, run:
regression_single_model_performance = Dashboard(tabs=[RegressionPerformanceTab])
regression_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
To generate the Classification Model Performance report, run:
classification_performance_report = Dashboard(tabs=[ClassificationPerformanceTab])
classification_performance_report.calculate(reference_data, current_data, column_mapping = column_mapping)
For Probabilistic Classification Model Performance report, run:
classification_performance_report = Dashboard(tabs=[ProbClassificationPerformanceTab])
classification_performance_report.calculate(reference_data, current_data, column_mapping = column_mapping)
You can also generate either of the Classification reports for a single DataFrame. In this case, run:
classification_single_model_performance = Dashboard(tabs=[ClassificationPerformanceTab])
classification_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
or
prob_classification_single_model_performance = Dashboard(tabs=[ProbClassificationPerformanceTab])
prob_classification_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
Option 2: Profile
After installing the tool, import Evidently profile and required sections:
import pandas as pd
from sklearn import datasets
from evidently.model_profile import Profile
from evidently.profile_sections import DataDriftProfileSection
iris = datasets.load_iris()
iris_frame = pd.DataFrame(iris.data, columns = iris.feature_names)
To generate the Data Drift profile, run:
iris_data_drift_profile = Profile(sections=[DataDriftProfileSection])
iris_data_drift_profile.calculate(iris_frame, iris_frame, column_mapping = None)
iris_data_drift_profile.json()
To generate the Data Drift and the Categorical Target Drift profile, run:
iris_target_and_data_drift_profile = Profile(sections=[DataDriftProfileSection, CatTargetDriftProfileSection])
iris_target_and_data_drift_profile.calculate(iris_frame[:75], iris_frame[75:], column_mapping = None)
iris_target_and_data_drift_profile.json()
You can also generate a Regression Model Performance for a single DataFrame. In this case, run:
regression_single_model_performance = Profile(sections=[RegressionPerformanceProfileSection])
regression_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
To generate the Classification Model Performance profile, run:
classification_performance_profile = Profile(sections=[ClassificationPerformanceProfileSection])
classification_performance_profile.calculate(reference_data, current_data, column_mapping = column_mapping)
For Probabilistic Classification Model Performance profile, run:
classification_performance_report = Profile(sections=[ProbClassificationPerformanceProfileSection])
classification_performance_report.calculate(reference_data, current_data, column_mapping = column_mapping)
You can also generate either of the Classification profiles for a single DataFrame. In this case, run:
classification_single_model_performance = Profile(sections=[ClassificationPerformanceProfileSection])
classification_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
or
prob_classification_single_model_performance = Profile(sections=[ProbClassificationPerformanceProfileSection])
prob_classification_single_model_performance.calculate(reference_data, None, column_mapping=column_mapping)
Terminal
You can generate HTML reports or JSON profiles directly from the bash shell. To do this, prepare your data as two csv files. In case you run one of the performance reports, you can have only one file. The first one should include your reference data, the second - current production data. The structure of both datasets should be identical.
To generate a HTML report, run the following command in bash:
python -m evidently calculate dashboard --config config.json
--reference reference.csv --current current.csv --output output_folder --report_name output_file_name
To generate a JSON profile, run the following command in bash:
python -m evidently calculate profile --config config.json
--reference reference.csv --current current.csv --output output_folder --report_name output_file_name
Here:
referenceis the path to the reference data,currentis the path to the current data,outputis the path to the output folder,report_nameis name of the output file,configis the path to the configuration file,pretty_printto print the JSON profile with indents (for profile only).
Currently, you can choose the following Tabs or Sections:
data_driftto estimate the data drift,num_target_driftto estimate target drift for numerical target,cat_target_driftto estimate target drift for categorical target,classification_performanceto explore the performance of a classification model,prob_classification_performanceto explore the performance of a probabilistic classification model,regression_performanceto explore the performance of a regression model.
To configure a report or a profile you need to create the config.json file. This file configures the way of reading your input data and the type of the report.
Here is an example of a simple configuration for a report, where we have comma separated csv files with headers and there is no date column in the data.
Dashboard:
{
"data_format": {
"separator": ",",
"header": true,
"date_column": null
},
"column_mapping" : {},
"dashboard_tabs": ["cat_target_drift"]
}
Profile:
{
"data_format": {
"separator": ",",
"header": true,
"date_column": null
},
"column_mapping" : {},
"profile_sections": ["data_drift"],
"pretty_print": true
}
Here is an example of a more complicated configuration, where we have comma separated csv files with headers and datetime column. We also specified the column_mapping dictionary to add information about datetime, target and numerical_features.
Dashboard:
{
"data_format": {
"separator": ",",
"header": true,
"date_column": "datetime"
},
"column_mapping" : {
"datetime":"datetime",
"target":"target",
"numerical_features": ["mean radius", "mean texture", "mean perimeter",
"mean area", "mean smoothness", "mean compactness", "mean concavity",
"mean concave points", "mean symmetry"]},
"dashboard_tabs": ["cat_target_drift"],
"sampling": {
"reference": {
"type": "none"
},
"current": {
"type": "nth",
"n": 2
}
}
}
Profile:
{
"data_format": {
"separator": ",",
"header": true,
"date_column": null
},
"column_mapping" : {
"target":"target",
"numerical_features": ["mean radius", "mean texture", "mean perimeter",
"mean area", "mean smoothness", "mean compactness", "mean concavity",
"mean concave points", "mean symmetry"]},
"profile_sections": ["data_drift", "cat_target_drift"],
"pretty_print": true,
"sampling": {
"reference": {
"type": "none"
},
"current": {
"type": "random",
"ratio": 0.8
}
}
}
Telemetry
When you use Evidently in the command-line interface, we collect basic telemetry (starting from 0.1.21.dev0 version). It includes data on the environment (e.g. Python version) and usage (type of report or profile generated). You can read more about what we collect here.
You can opt-out from telemetry collection by setting the environment variable EVIDENTLY_DISABLE_TELEMETRY=1
Large datasets
As you can see from the above example, you can specify sampling parameters for large files. You can use different sampling strategies for reference and current data, or apply sampling only to one of the files. Currently we have 3 sampling types available:
none- there will be no sampling for the file,nth- each Nth row of the file will be taken. This option works together withnparameter (see the example with the Dashboard above)random- random sampling will be applied. This option works together withratioparameter (see the example with the Profile above)
Documentation
For more information, refer to a complete Documentation.
Examples
-
See Data Drift Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate .html file: Iris, Boston
-
See Numerical Target and Data Drift Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate file: Iris, Breast Cancer
-
See Categorical Target and Data Drift Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate file: Boston
-
See Regression Performance Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate file: Bike Sharing Demand
-
See Classification Performance Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate file: Iris
-
See Probabilistic Classification Performance Dashboard and Profile generation to explore the results both inside a Jupyter notebook and as a separate .html file: Iris, Breast Cancer
Stay updated
We will be releasing more reports soon. If you want to receive updates, follow us on Twitter, or sign up for our newsletter. You can also find more tutorials and explanations in our Blog. If you want to chat and connect, join our Discord community!
![[BUG] Broken links in the readme](https://avatars.githubusercontent.com/u/784901?v=4)
9 May 05, 2022
1.4k Jan 03, 2023
299 Dec 30, 2022
297 Jan 09, 2023
11.7k Dec 30, 2022
371 Dec 21, 2022
3 Feb 15, 2022
1.3k Jan 05, 2023
5 Nov 01, 2022
10 Apr 04, 2022
1 Oct 29, 2021
1.4k Jan 15, 2022
1.4k Jan 01, 2023
1 Dec 06, 2022
7 Dec 18, 2021
54 Nov 06, 2022
31 Nov 17, 2022
12 Aug 23, 2022