System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): N/A
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux CentOS 7
• TensorFlow Model Analysis installed from (source or binary): pypi binary
• TensorFlow Model Analysis version (use command below): tensorflow-model-analysis 0.14.0
• Python version: 3.6
• Jupyter Notebook version: 7.0.0
• Exact command to reproduce: N/A

Describe the problem

At Twitter, we primarily use the JupyterLab front-end for our notebook-based workflows. TFMA currently only supports running as an nbextension for the "Classic" Notebook UI - vs providing a labextension for e.g. JupyterLab.

Thus, consuming TFMA currently requires that our ML practitioners revert to the "Classic" Notebook UI which has largely been deprecated internally. It'd be great if TFMA could provide a JupyterLab plugin so that our users didn't have to switch UIs and interrupt their typical workflow.

Source code / logs

N/A

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): NO
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Google Colab
• TensorFlow Model Analysis installed from (source or binary): source
• TensorFlow Model Analysis version (use command below): 0.27.0
• Python version: 3.6.9
• Jupyter Notebook version: Google Colab
• Exact command to reproduce:

Describe the problem

Hi, i've been following the TFX Chicago Taxi Example (https://www.tensorflow.org/tfx/tutorials/tfx/components_keras#evaluator) to factor my TensorFlow code into the TFX framework.

However, for my use case, it's a multi-output keras model, where the model consumes a given input, and produces 2 outputs (both being multi-class).

If i ran the evaluator component with just 1 output (e.g.: disable the other output in my model) , it works fine and i can run tfma.run_model_analysis without an issue.

However, reverting to my multi-output model, running the evaluator component throws up an error.

Model - output_0 has 5 classes, and output_1 has 8 classes to predict >>

signature_def['serving_raw']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['CREDIT'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1)
        name: serving_raw_CREDIT:0
    inputs['DEBIT'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1)
        name: serving_raw_DEBIT:0
    inputs['DESCRIPTION'] tensor_info:
        dtype: DT_STRING
        shape: (-1, -1)
        name: serving_raw_DESCRIPTION:0
    inputs['TRADEDATE'] tensor_info:
        dtype: DT_STRING
        shape: (-1, -1)
        name: serving_raw_TRADEDATE:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['output_0'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 5)
        name: StatefulPartitionedCall_2:0
    outputs['output_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 8)

Eval_Config >>

eval_config = tfma.EvalConfig(
    model_specs=[     
        tfma.ModelSpec(label_key='my_label_key')
    ],
    metrics_specs=[
        tfma.MetricsSpec(
            metrics=[
                tfma.MetricConfig(class_name = 'SparseCategoricalAccuracy', 
                                  threshold=tfma.MetricThreshold(
                                      value_threshold=tfma.GenericValueThreshold(lower_bound={'value': 0.5}),
                                      change_threshold=tfma.GenericChangeThreshold(
                                          direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                          absolute={'value': -1e-10}))),
                tfma.MetricConfig(class_name = 'MultiClassConfusionMatrixPlot'),
                tfma.MetricConfig(class_name = "Precision"),
                tfma.MetricConfig(class_name = "Recall")
            ], 
            output_names =['output_0']
        ),
         tfma.MetricsSpec(
            metrics=[
                tfma.MetricConfig(class_name = 'SparseCategoricalAccuracy', 
                                  threshold=tfma.MetricThreshold(
                                      value_threshold=tfma.GenericValueThreshold(lower_bound={'value': 0.5}),
                                      change_threshold=tfma.GenericChangeThreshold(
                                          direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                          absolute={'value': -1e-10}))),
                tfma.MetricConfig(class_name = 'MultiClassConfusionMatrixPlot'),
                tfma.MetricConfig(class_name = "Precision"),
                tfma.MetricConfig(class_name = "Recall")
            ], 
            output_names =['output_1']
         )
    ],
    slicing_specs=[
        tfma.SlicingSpec(),
    ])

Running tfma.run_model_analysis using the above eval_config,

keras_model_path = os.path.join(trainer.outputs['model'].get()[0].uri,'serving_model_dir') # gets the model from the trainer stage
keras_eval_shared_model = tfma.default_eval_shared_model(
    eval_saved_model_path=keras_model_path,
    eval_config=eval_config)

keras_output_path = os.path.join(os.getcwd(), 'keras2')
tfrecord_file = '/tmp/tfx-interactive-2021-02-09T06_02_48.210135-95bh38cw/Transform/transformed_examples/5/train/transformed_examples-00000-of-00001.gz'
# Run TFMA
keras_eval_result = tfma.run_model_analysis(
    eval_shared_model=keras_eval_shared_model,
    eval_config=eval_config,
    data_location=tfrecord_file,
    output_path=keras_output_path)

I get an error message of the below >>

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow_model_analysis/model_util.py in process(self, element)
    667     try:
--> 668       result = self._batch_reducible_process(element)
    669       self._batch_size.update(batch_size)

118 frames
ValueError: could not broadcast input array from shape (5) into shape (1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
ValueError: could not broadcast input array from shape (5) into shape (1)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a, dtype, order)
     81 
     82     """
---> 83     return array(a, dtype, copy=False, order=order)
     84 
     85 

ValueError: could not broadcast input array from shape (5) into shape (1) [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/ExtractPredictions/Predict']

I've tried to find code examples of multi-output eval_config but haven't come across one yet.

Following the documentation, i've arrived at what i think the eval_config should be for a multi-output model - however is it set up correctly given the error message?

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): N/A
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): N/A
• TensorFlow Model Analysis installed from (source or binary): N/A
• TensorFlow Model Analysis version (use command below): N/A
• Python version: N/A
• Jupyter Notebook version: N/A
• Exact command to reproduce: N/A

Describe the problem

The HEAD version of TFMA seems to be missing some definitions for third-party dependencies. The easy-to-fix one is ProtoBuf:

ERROR: error loading package 'tensorflow_model_analysis/proto': Extension file not found. Unable to load package for '@protobuf_bzl//:protobuf.bzl': The repository could not be resolved

which can be fixed by changing the load call slightly

diff --git a/tensorflow_model_analysis/proto/BUILD b/tensorflow_model_analysis/proto/BUILD
index af3386c..af787da 100644
--- a/tensorflow_model_analysis/proto/BUILD
+++ b/tensorflow_model_analysis/proto/BUILD
@@ -2,7 +2,7 @@ licenses(["notice"])  # Apache 2.0

 package(default_visibility = ["//visibility:public"])

-load("@protobuf_bzl//:protobuf.bzl", "py_proto_library")
+load("@com_google_protobuf//:protobuf.bzl", "py_proto_library")

The other one is a bit more difficult since the BUILD file for third_party/py/typing is truly missing from the repo

ERROR: [...]/model-analysis/tensorflow_model_analysis/slicer/BUILD:3:1: no such package 'third_party/py/typing': BUILD file not found on package path and referenced by '//tensorflow_model_analysis/slicer:slicer'

Finally, some of the TFMA BUILD files reference third_party/py/numpy which is missing as well.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): N/A
• TensorFlow Model Analysis installed from (source or binary): PyPI
• TensorFlow Model Analysis version (use command below): 0.6.0
• Python version: 2.7
• Jupyter Notebook version: N/A
• Exact command to reproduce:

import tensorflow_model_analysis as tfma
from ipywidgets.embed import embed_minimal_html

analysis_path = 'gs://<TFMA_OUTPUT_DIRECTORY>'
result = tfma.load_eval_result(output_path=analysis_path)
slicing_metrics_view = tfma.view.render_slicing_metrics(result)
embed_minimal_html('tfma_export.html', views=[slicing_metrics_view], title='Slicing Metrics')

Describe the problem

Jupyter notebook widgets support embedding the widget in a static HTML file that can be loaded outside of a Jupyter notebook (see here for details).

This almost works for the TFMA widgets, except that the tfma_widget_js.js file assumes that it is running in the context of a notebook page, and tries to load the vulcanized_template.html file from the notebook server (which won't exist in the case of a static HTML file).

Thus, I am filing a feature request to put that file on a CDN somewhere, and to teach the tfma_widget_js.js code to load it from the CDN if necessary.

Source code / logs

Here is the line in the JS file where it tries to load the vulcanized_template.html file from the notebook server.

The simplest way I could suggest to enhance this would be to check of the data-base-url document attribute is null (indicating that the code is running outside of a notebook), and in that case have the __webpack_require__.p location resolve to a CDN URL.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): N/A
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Tested in MacOS, would like support for Linux CentOS 7
• TensorFlow Model Analysis installed from (source or binary): PyPI binary
• TensorFlow Model Analysis version (use command below): 0.27.0
• Python version: 3.6
• Jupyter Notebook version: Using jupyterlab v3.0.11
• Exact command to reproduce: Installing TFMA

$  pip install tensorflow_model_analysis==0.27.0
$ jupyter labextension install [email protected]
$ pip install jupyterlab_widgets
$ jupyter labextension list
JupyterLab v3.0.11
/Users/mwakabayashi/opt/anaconda3/envs/jupyterlab3/share/jupyter/labextensions
        @jupyter-widgets/jupyterlab-manager v3.0.0 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
   app dir: /Users/mwakabayashi/opt/anaconda3/envs/jupyterlab3/share/jupyter/lab
        tensorflow_model_analysis v0.27.0 enabled OK

Running TFMA basic notebook

1. jupyter lab
2. Ran the TFMA basic notebook with tensorflow==2.3.0.

Describe the problem

Jupyterlab 3.0 was released in January 2021. Would it be possible to get v3 support (install TFMA as a prebuilt extension) soon? At Twitter, we'd like to migrate to Jupyterlab 3 from 2, but we can't without v3 support for TFMA.

Source code / logs

Error after running render_slicing_metrics function.

In Chrome developer console

Issue

I'm part of the team supporting TFX/Kubeflow Pipelines at Spotify, we are currently upgrading our internal stack to tfx==0.22.1 and tensorflow-model-analysis==0.22.2.

We can successfully evaluate an Estimator-based model with the open-source evaluator component. Unfortunately, when using a Keras-based model, the Beam evaluation pipeline running on Dataflow fails with the following error:

Traceback (most recent call last):
  File "apache_beam/runners/common.py", line 961, in apache_beam.runners.common.DoFnRunner.process
  File "apache_beam/runners/common.py", line 726, in apache_beam.runners.common.PerWindowInvoker.invoke_process
  File "apache_beam/runners/common.py", line 812, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
  File "apache_beam/runners/common.py", line 1122, in apache_beam.runners.common._OutputProcessor.process_outputs
  File "apache_beam/runners/worker/operations.py", line 195, in apache_beam.runners.worker.operations.SingletonConsumerSet.receive
  File "apache_beam/runners/worker/operations.py", line 949, in apache_beam.runners.worker.operations.PGBKCVOperation.process
  File "apache_beam/runners/worker/operations.py", line 978, in apache_beam.runners.worker.operations.PGBKCVOperation.process
  File "/usr/local/lib/python3.6/dist-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py", line 356, in add_input
    result = c.add_input(a, get_combiner_input(elements[0], i))
  File "/usr/local/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py", line 551, in add_input
    flatten=self._class_weights is not None)):
  File "/usr/local/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py", line 264, in to_label_prediction_example_weight
    sub_key, inputs))
ValueError: no value provided for label: model_name=, output_name=, sub_key=None, StandardMetricInputs=StandardMetricInputs(label=None, prediction=0.025542974, example_weight=None, features=None)
This may be caused by a configuration error (i.e. label, and/or prediction keys were not specified) or an error in the pipeline.

Have you ever faced this issue?

Additional context

• We run a version of the Chicago taxi example pipeline
• Keras model was trained using a copy of the GenericExecutor
• The model's code was copied from TFX's tutorial
• The following eval config is passed to the Evaluator component:

import tensorflow_model_analysis as tfma
from google.protobuf.wrappers_pb2 import BoolValue
eval_config = tfma.EvalConfig(
    model_specs=[
      tfma.ModelSpec(model_type=tfma.constants.TF_KERAS, label_key="tips")
    ],
    slicing_specs=[
        tfma.SlicingSpec(),
    ],
    options=tfma.Options(include_default_metrics=BoolValue(value=True)),
)

System information

• Have I written custom code: Yes
• TensorFlow Model Analysis installed from: binary via pip
• TensorFlow Model Analysis version (use command below): 0.22.2
• Python version: 3.6.9

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS Catalina
• TensorFlow Model Analysis installed from (source or binary): pypi
• TensorFlow Model Analysis version (use command below): 0.22.1
• Python version: 3.7.5
• Jupyter Notebook version: 1.0.0

Describe the problem

I have trained a Keras model (not estimator) with the following serving signature:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['examples'] tensor_info:
        dtype: DT_STRING
        shape: (-1)
        name: serving_default_examples:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['mu'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:0
    outputs['sigma'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 1)
        name: StatefulPartitionedCall_1:1
  Method name is: tensorflow/serving/predict

The weights are updated using a custom training loop with gradient tape, instead of the model.fit method, before the model is exported as a saved_model. As I am unable to get TFMA to work without first compiling the model, I compile the model while specifying a set of custom Keras metrics:

model.compile(metrics=custom_keras_metrics) # each custom metric inherits from keras.Metric
custom_training_loop(model)
model.save("path/to/saved_model", save_format="tf")

I would like to evaluate this model using TFMA, so I first initialise an eval shared model as follows:

eval_config = tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key="my_label_key")],
    slicing_specs=[tfma.SlicingSpec()] # empty slice refers to the entire dataset
)
eval_shared_model = tfma.default_eval_shared_model("path/to/saved_model", eval_config=eval_config)

However, when I try to run model analysis:

eval_results = tfma.run_model_analysis(
    eval_shared_model=eval_shared_model,
    data_location="path/to/test/tfrecords*",
    file_format="tfrecords"
)

I am faced with the following error:

ValueError          Traceback (most recent call last)
<ipython-input-156-f9a9684a6797> in <module>
      2     eval_shared_model=eval_shared_model,
      3     data_location="tfma/test_raw-*",
----> 4     file_format="tfrecords"
      5 )

~/.pyenv/versions/miniconda3-4.3.30/envs/tensorflow/lib/python3.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in run_model_analysis(eval_shared_model, eval_config, data_location, file_format, output_path, extractors, evaluators, writers, pipeline_options, slice_spec, write_config, compute_confidence_intervals, min_slice_size, random_seed_for_testing, schema)
   1204 
   1205   if len(eval_config.model_specs) <= 1:
-> 1206     return load_eval_result(output_path)
   1207   else:
   1208     results = []

~/.pyenv/versions/miniconda3-4.3.30/envs/tensorflow/lib/python3.7/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in load_eval_result(output_path, model_name)
    383       metrics_and_plots_serialization.load_and_deserialize_metrics(
    384           path=os.path.join(output_path, constants.METRICS_KEY),
--> 385           model_name=model_name))
    386   plots_proto_list = (
    387       metrics_and_plots_serialization.load_and_deserialize_plots(

~/.pyenv/versions/miniconda3-4.3.30/envs/tensorflow/lib/python3.7/site-packages/tensorflow_model_analysis/writers/metrics_and_plots_serialization.py in load_and_deserialize_metrics(path, model_name)
    180       raise ValueError('Fail to find metrics for model name: %s . '
    181                        'Available model names are [%s]' %
--> 182                        (model_name, ', '.join(keys)))
    183 
    184     result.append((

ValueError: Fail to find metrics for model name: None . Available model names are []

Why is TFMA raising this exception, and where should I begin debugging this error? I tried specifying the model names manually (which should not be required since I'm only using one model), but that did not seem to help either. I tried tracing the source code and it seems this happens when TFMA tries to load the eval result generated by the PTransform.

Referencing this issue: #50. We ran into the exact same issue not allowing the extension to be enabled in the notebook. When installing the extension, the static directory is looked up relative to the package location. When building a binary distribution, no static directory was created relative to the tensorflow_model_analysis package location.

To reproduce prior:

python setup.py bdist_wheel
unzip -l tensorflow_model_analysis-0.15.0.dev0-py2-none-any.whl | grep "tensorflow_model_analysis/static/*.js"

Using setuptools instead of distutils resolved the issue.

What's the timeline for supporting Python 3?

TensorFlow only runs on Python 3.5 and 3.6 on Windows. So those of us who work on Windows machines have a harder time trying out TFMA.

#14 # System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS Linux 7 (Core)
• TensorFlow Model Analysis installed from (source or binary): Binary
• TensorFlow Model Analysis version (use command below):0.24.3
• Python version:3.6.9
• Jupyter Notebook version:6.0.1
• Exact command to reproduce:

Describe the problem

I am following the tfx tutorial https://www.tensorflow.org/tfx/tutorials/tfx/components_keras, but with multilabel data.(number of categories is 5). Here is the output from example_gen

{
 'Text': array([b'Football fans looking forward to seeing the renewal of the rivalry between Cristiano Ronaldo and Lionel Messi were made to wait a while longer after the Portuguese forward was forced to miss Juventus' Champions League tie against Barcelona on Wednesday.'],dtype=object),
 'Headline': array([b"Lionel Messi scores as Cristiano Ronaldo misses Barcelona's victory over Juventus."], dtype=object),
 'categories': array([b'Sports'], dtype=object), 
}

{
 'Text': array([b'COVID-19 has changed fan behavior and accelerated three to five years of technology adoption into six months'],dtype=object),
 'Headline': array([b"How Technology Is Improving Fan Transactions at Sports Venues"], dtype=object),
 'categories': array([b'Sports', b'Science and Technology'], dtype=object), 
}

Output from tf transform:

{
 'Text': array([b'Football fans looking forward to seeing the renewal of the rivalry between Cristiano Ronaldo and Lionel Messi were made to wait a while longer after the Portuguese forward was forced to miss Juventus' Champions League tie against Barcelona on Wednesday.'],dtype=object),
 'Headline': array([b"Lionel Messi scores as Cristiano Ronaldo misses Barcelona's victory over Juventus."], dtype=object),
 'categories': array([1., 0., 0., 0., 0.], dtype=object), 
}

{
 'Text_xf': array([b'COVID-19 has changed fan behavior and accelerated three to five years of technology adoption into six months'],dtype=object),
 'Headline_xf': array([b"How Technology Is Improving Fan Transactions at Sports Venues"], dtype=object),
 'categories_xf': array([1., 1., 0., 0., 0.'], dtype=object), 
}

I have trained the model using Trainer and then I want to use TFMA

metrics = [
    tf.keras.metrics.Recall(name='recall', top_k=3),
]
metrics_specs = tfma.metrics.specs_from_metrics(metrics)

eval_config=tfma.EvalConfig(
    model_specs=[tfma.ModelSpec(label_key="categories")],
    slicing_specs=[tfma.SlicingSpec()],
    metrics_specs=metrics_specs,
    
)

evaluator = Evaluator(
    examples=example_gen.outputs['examples'],
    model=trainer.outputs['model'],
    baseline_model=model_resolver.outputs['model'],
    eval_config=eval_config
)
context.run(evaluator)

logs

WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27adc9d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:6 out of the last 6 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27abc6a8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:7 out of the last 7 calls to <function recreate_function.<locals>.restored_function_body at 0x7f5a27abc0d0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:8 out of the last 8 calls to <function recreate_function.<locals>.restored_function_body at 0x7f59386e9c80> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
WARNING:tensorflow:9 out of the last 9 calls to <function recreate_function.<locals>.restored_function_body at 0x7f59386d39d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py in add_input(self, accumulator, element)
    340     for i, (c, a) in enumerate(zip(self._combiners, accumulator)):
--> 341       result = c.add_input(a, get_combiner_input(elements[0], i))
    342       for e in elements[1:]:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in add_input(self, accumulator, element)
    576         if self._is_top_k() and label.shape != prediction.shape:
--> 577           label = metric_util.one_hot(label, prediction)
    578         accumulator.add_input(i, label, prediction, example_weight)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py in one_hot(tensor, target)
    703   # indexing the -1 and then removing it after.
--> 704   tensor = np.delete(np.eye(target.shape[-1] + 1)[tensor], -1, axis=-1)
    705   return tensor.reshape(target.shape)

IndexError: arrays used as indices must be of integer (or boolean) type

During handling of the above exception, another exception occurred:

IndexError                                Traceback (most recent call last)
<ipython-input-31-952eda92fce9> in <module>
      5     eval_config=eval_config
      6 )
----> 7 context.run(evaluator)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/experimental/interactive/interactive_context.py in run_if_ipython(*args, **kwargs)
     65       # __IPYTHON__ variable is set by IPython, see
     66       # https://ipython.org/ipython-doc/rel-0.10.2/html/interactive/reference.html#embedding-ipython.
---> 67       return fn(*args, **kwargs)
     68     else:
     69       absl.logging.warning(

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/experimental/interactive/interactive_context.py in run(self, component, enable_cache, beam_pipeline_args)
    180         telemetry_utils.LABEL_TFX_RUNNER: runner_label,
    181     }):
--> 182       execution_id = launcher.launch().execution_id
    183 
    184     return execution_result.ExecutionResult(

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/launcher/base_component_launcher.py in launch(self)
    203                          execution_decision.input_dict,
    204                          execution_decision.output_dict,
--> 205                          execution_decision.exec_properties)
    206 
    207     absl.logging.info('Running publisher for %s',

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/orchestration/launcher/in_process_component_launcher.py in _run_executor(self, execution_id, input_dict, output_dict, exec_properties)
     65         executor_context)  # type: ignore
     66 
---> 67     executor.Do(input_dict, output_dict, exec_properties)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tfx/components/evaluator/executor.py in Do(self, input_dict, output_dict, exec_properties)
    258            output_path=output_uri,
    259            slice_spec=slice_spec,
--> 260            tensor_adapter_config=tensor_adapter_config))
    261     logging.info('Evaluation complete. Results written to %s.', output_uri)
    262 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/pipeline.py in __exit__(self, exc_type, exc_val, exc_tb)
    553     try:
    554       if not exc_type:
--> 555         self.result = self.run()
    556         self.result.wait_until_finish()
    557     finally:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/pipeline.py in run(self, test_runner_api)
    532         finally:
    533           shutil.rmtree(tmpdir)
--> 534       return self.runner.run_pipeline(self, self._options)
    535     finally:
    536       shutil.rmtree(self.local_tempdir, ignore_errors=True)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_pipeline(self, pipeline, options)
    174 
    175     self._latest_run_result = self.run_via_runner_api(
--> 176         pipeline.to_runner_api(default_environment=self._default_environment))
    177     return self._latest_run_result
    178 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_via_runner_api(self, pipeline_proto)
    184     # TODO(pabloem, BEAM-7514): Create a watermark manager (that has access to
    185     #   the teststream (if any), and all the stages).
--> 186     return self.run_stages(stage_context, stages)
    187 
    188   @contextlib.contextmanager

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_stages(self, stage_context, stages)
    342           stage_results = self._run_stage(
    343               runner_execution_context,
--> 344               bundle_context_manager,
    345           )
    346           monitoring_infos_by_stage[stage.name] = (

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_stage(self, runner_execution_context, bundle_context_manager)
    521               input_timers,
    522               expected_timer_output,
--> 523               bundle_manager)
    524 
    525       final_result = merge_results(last_result)

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_bundle(self, runner_execution_context, bundle_context_manager, data_input, data_output, input_timers, expected_timer_output, bundle_manager)
    559 
    560     result, splits = bundle_manager.process_bundle(
--> 561         data_input, data_output, input_timers, expected_timer_output)
    562     # Now we collect all the deferred inputs remaining from bundle execution.
    563     # Deferred inputs can be:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    943     with thread_pool_executor.shared_unbounded_instance() as executor:
    944       for result, split_result in executor.map(execute, zip(part_inputs,  # pylint: disable=zip-builtin-not-iterating
--> 945                                                             timer_inputs)):
    946         split_result_list += split_result
    947         if merged_result is None:

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in result_iterator()
    584                     # Careful not to keep a reference to the popped future
    585                     if timeout is None:
--> 586                         yield fs.pop().result()
    587                     else:
    588                         yield fs.pop().result(end_time - time.monotonic())

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in result(self, timeout)
    430                 raise CancelledError()
    431             elif self._state == FINISHED:
--> 432                 return self.__get_result()
    433             else:
    434                 raise TimeoutError()

~/anaconda3/envs/tf2/lib/python3.6/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/utils/thread_pool_executor.py in run(self)
     42       # If the future wasn't cancelled, then attempt to execute it.
     43       try:
---> 44         self._future.set_result(self._fn(*self._fn_args, **self._fn_kwargs))
     45       except BaseException as exc:
     46         # Even though Python 2 futures library has #set_exection(),

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in execute(part_map_input_timers)
    939           input_timers,
    940           expected_output_timers,
--> 941           dry_run)
    942 
    943     with thread_pool_executor.shared_unbounded_instance() as executor:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    839             process_bundle_descriptor.id,
    840             cache_tokens=[next(self._cache_token_generator)]))
--> 841     result_future = self._worker_handler.control_conn.push(process_bundle_req)
    842 
    843     split_results = []  # type: List[beam_fn_api_pb2.ProcessBundleSplitResponse]

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py in push(self, request)
    351       self._uid_counter += 1
    352       request.instruction_id = 'control_%s' % self._uid_counter
--> 353     response = self.worker.do_instruction(request)
    354     return ControlFuture(request.instruction_id, response)
    355 

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in do_instruction(self, request)
    481       # E.g. if register is set, this will call self.register(request.register))
    482       return getattr(self, request_type)(
--> 483           getattr(request, request_type), request.instruction_id)
    484     else:
    485       raise NotImplementedError

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in process_bundle(self, request, instruction_id)
    516         with self.maybe_profile(instruction_id):
    517           delayed_applications, requests_finalization = (
--> 518               bundle_processor.process_bundle(instruction_id))
    519           monitoring_infos = bundle_processor.monitoring_infos()
    520           monitoring_infos.extend(self.state_cache_metrics_fn())

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_bundle(self, instruction_id)
    981           elif isinstance(element, beam_fn_api_pb2.Elements.Data):
    982             input_op_by_transform_id[element.transform_id].process_encoded(
--> 983                 element.data)
    984 
    985       # Finish all operations.

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_encoded(self, encoded_windowed_values)
    217       decoded_value = self.windowed_coder_impl.decode_from_stream(
    218           input_stream, True)
--> 219       self.output(decoded_value)
    220 
    221   def monitoring_infos(self, transform_id, tag_to_pcollection_id):

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process_with_sized_restriction()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.ConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/future/utils/__init__.py in raise_with_traceback(exc, traceback)
    444         if traceback == Ellipsis:
    445             _, _, traceback = sys.exc_info()
--> 446         raise exc.with_traceback(traceback)
    447 
    448 else:

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.PGBKCVOperation.process()

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/evaluators/metrics_and_plots_evaluator_v2.py in add_input(self, accumulator, element)
    339     results = []
    340     for i, (c, a) in enumerate(zip(self._combiners, accumulator)):
--> 341       result = c.add_input(a, get_combiner_input(elements[0], i))
    342       for e in elements[1:]:
    343         result = c.add_input(result, get_combiner_input(e, i))

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py in add_input(self, accumulator, element)
    575         # Keras requires non-sparse keys for top_k calcuations.
    576         if self._is_top_k() and label.shape != prediction.shape:
--> 577           label = metric_util.one_hot(label, prediction)
    578         accumulator.add_input(i, label, prediction, example_weight)
    579     if (accumulator.len_inputs() >= self._batch_size or

~/anaconda3/envs/tf2/lib/python3.6/site-packages/tensorflow_model_analysis/metrics/metric_util.py in one_hot(tensor, target)
    702   # the row. The following handles -1 values by adding an additional column for
    703   # indexing the -1 and then removing it after.
--> 704   tensor = np.delete(np.eye(target.shape[-1] + 1)[tensor], -1, axis=-1)
    705   return tensor.reshape(target.shape)
    706 

IndexError: arrays used as indices must be of integer (or boolean) type [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/ComputePerSlice/ComputeUnsampledMetrics/CombinePerSliceKey/WindowIntoDiscarding']

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.13.6
• TensorFlow Model Analysis installed from (source or binary): binary
• TensorFlow Model Analysis version (use command below): 0.9.0
• Python version: 2.7.14
• Jupyter Notebook version: 1.0.0
• Exact command to reproduce: Follow Chicago Taxi Example (local example) tutorial.

Describe the problem

I ran Chicago Taxi Example notebook on Firefox and the TF Model Analysis interactive widget didn't show up. At first I thought there was a problem when I installed and enabled tensorflow_model_analysis Jupyter nbextension. It turned out when I opened the notebook in Chrome, it worked perfectly.

Source code / logs

Firefox' console showed that tfma_widget_js is successfully loaded:

Use of Mutation Events is deprecated. Use MutationObserver instead. jquery.min.js:2
actions jupyter-notebook:find-and-replace does not exist, still binding it in case it will be defined later... menubar.js:277
accessing "actions" on the global IPython/Jupyter is not recommended. Pass it to your objects contructors at creation time main.js:208
Loaded moment locale en bidi.js:19
load_extensions 
Arguments { 0: "jupyter-js-widgets/extension", 1: "tfma_widget_js/extension", … }
utils.js:60
Loading extension: tfma_widget_js/extension utils.js:37
Session: kernel_created (fedc5d8c-5b4b-4b18-9029-685745010dd4) session.js:54
Starting WebSockets: ws://localhost:8888/api/kernels/fd4631da-a6de-4077-bc96-b52c0fa6a0e9 kernel.js:459
Loading extension: jupyter-js-widgets/extension utils.js:37
Kernel: kernel_connected (fd4631da-a6de-4077-bc96-b52c0fa6a0e9) kernel.js:103
Kernel: kernel_ready (fd4631da-a6de-4077-bc96-b52c0fa6a0e9) kernel.js:103 

Comparison to Chrome's console:

load_extensions Arguments(2) ["jupyter-js-widgets/extension", "tfma_widget_js/extension", callee: (...), Symbol(Symbol.iterator): ƒ]
bidi.js:19 Loaded moment locale en
session.js:54 Session: kernel_created (fedc5d8c-5b4b-4b18-9029-685745010dd4)
kernel.js:459 Starting WebSockets: ws://localhost:8888/api/kernels/fd4631da-a6de-4077-bc96-b52c0fa6a0e9
utils.js:37 Loading extension: tfma_widget_js/extension
kernel.js:103 Kernel: kernel_connected (fd4631da-a6de-4077-bc96-b52c0fa6a0e9)
kernel.js:103 Kernel: kernel_ready (fd4631da-a6de-4077-bc96-b52c0fa6a0e9)
utils.js:37 Loading extension: jupyter-js-widgets/extension
[Deprecation] Styling master document from stylesheets defined in HTML Imports is deprecated. Please refer to https://goo.gl/EGXzpw for possible migration paths.

https://github.com/tensorflow/model-analysis/blob/b4f2e9fe2733a8ee4d6a129facf23efdc4b162ba/setup.py#L297

Numpy >=1.20 is easy to install on Apple silicon, because it's precompiled, but 1.19 fails unless openblas is set up on the system (which it will not be for most users). (See this NumPy issue.)

See also this issue in scipy where they moved away from 1.19 because it fails to build on Apple Silicon. https://github.com/scipy/oldest-supported-numpy/issues/19

Recent release of TFMA (0.42 - 0.43 - master) changed the DoFn argument from preprocessor https://github.com/tensorflow/model-analysis/blob/a5c4c709e733bffe10038ee43b07704883d843b1/tensorflow_model_analysis/metrics/metric_types.py#L396 to preprocessors https://github.com/tensorflow/model-analysis/blob/e1c34a8b434440efa32d6617bc922a412c77b924/tensorflow_model_analysis/metrics/metric_types.py#L440

This breaking change is not reflected in the example for custom metrics here.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20.04.2 LTS Linux 5.4.0-65-generic
• TensorFlow Model Analysis installed from (source or binary): pip install tensorflow-model-analysis
• TensorFlow Model Analysis version (use command below): 0.41.1
• Python version: Python 3.8.5
• Jupyter Notebook version: NA

Describe the problem

I am currently trying to get tfma.analyze_raw_data to work with MultiClassConfusionMatrixPlot which has multiple prediction values per record. Is this not supported? I will be happy to provide any further details or run any further tests.

Details

Currently tfma.analyze_raw_data does not seem to work with metrics for multi classification tasks (e.g. tfma.metrics.MultiClassConfusionMatrixPlot). However, I do not see this limitation documented anywhere.

The prediction column for a multi classification column will be a series of whose values are a list or array (e.g.,. pd.DataFrame({'predictions': [[0.2, .3, .5]], 'label': [1]}))

The tfma.analyze_raw_data funciton uses tfx_bsl.arrow.DataFrameToRecordBatch to convert a Pandas DataFrame to Arrow RecordBatch. The problem, however, is that it encodes columns with the dtype of object as a pyarrow.Binary. Since a column that has lists or arrays as values has a dtype of object, these columns are being encoded as a pyarrow.Binary instead of the relevant pyarrow list-like type.

Source code / logs

import tensorflow_model_analysis as tfma
from google.protobuf import text_format
import pandas as pd

eval_config = text_format.Parse("""
  ## Model information
  model_specs {
    label_key: "label",
    prediction_key: "predictions"
  }

  ## Post training metric information. These will be merged with any built-in
  ## metrics from training.
  metrics_specs {
    metrics { class_name: "MultiClassConfusionMatrixPlot" }
  }
  
  ## Slicing information
  slicing_specs {}  # overall slice
""", tfma.EvalConfig())

df = pd.DataFrame({'predictions': [[0.2, .3, .5]], 'label': [1]})
tfma.analyze_raw_data(df, eval_config)

Error

---------------------------------------------------------------------------
ArrowTypeError                            Traceback (most recent call last)
/tmp/ipykernel_206830/3947320198.py in <cell line: 23>()
     21 
     22 df = pd.DataFrame({'predictions': [[0.2, .3, .5]], 'label': [1]})
---> 23 tfma.analyze_raw_data(df, eval_config)

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/tensorflow_model_analysis/api/model_eval_lib.py in analyze_raw_data(data, eval_config, output_path, add_metric_callbacks)
   1511 
   1512   arrow_data = table_util.CanonicalizeRecordBatch(
-> 1513       table_util.DataFrameToRecordBatch(data))
   1514   beam_data = beam.Create([arrow_data])
   1515 

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/tfx_bsl/arrow/table_util.py in DataFrameToRecordBatch(dataframe)
    122       continue
    123     arrow_fields.append(pa.field(col_name, arrow_type))
--> 124   return pa.RecordBatch.from_pandas(dataframe, schema=pa.schema(arrow_fields))
    125 
    126 

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/table.pxi in pyarrow.lib.RecordBatch.from_pandas()

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
    592 
    593     if nthreads == 1:
--> 594         arrays = [convert_column(c, f)
    595                   for c, f in zip(columns_to_convert, convert_fields)]
    596     else:

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py in <listcomp>(.0)
    592 
    593     if nthreads == 1:
--> 594         arrays = [convert_column(c, f)
    595                   for c, f in zip(columns_to_convert, convert_fields)]
    596     else:

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    579             e.args += ("Conversion failed for column {!s} with type {!s}"
    580                        .format(col.name, col.dtype),)
--> 581             raise e
    582         if not field_nullable and result.null_count > 0:
    583             raise ValueError("Field {} was non-nullable but pandas column "

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    573 
    574         try:
--> 575             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
    576         except (pa.ArrowInvalid,
    577                 pa.ArrowNotImplementedError,

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_array()

/localdisk/twilbers/src/repos/xai-tools/model_card_gen/.venv/lib/python3.8/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowTypeError: ("Expected bytes, got a 'list' object", 'Conversion failed for column predictions with type object')

Temporary fix

If I change/patch tfx_bsl.arrow.DataFrameToRecordBatch as follows, it seems to work, but I doubt this is a solution.

def DataFrameToRecordBatch(dataframe):
    arrays = []
    for col_name, col_type in zip(dataframe.columns, dataframe.dtypes):
        arrow_type = None
        if col_type.kind != 'O':
            arrow_type = NumpyKindToArrowType(col_type.kind)
        arrays.append(pa.array(dataframe[col_name].values.tolist(), type=arrow_type))
    return pa.RecordBatch.from_arrays(arrays,  names=dataframe.columns)

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): GCP Dataflow Apache Beam Python 3.7 SDK 2.39.0
• TensorFlow Model Analysis installed from (source or binary): binary
• TensorFlow Model Analysis version (use command below): 0.33
• Python version: 3.7
• Jupyter Notebook version: Jupyter lab 3.2.8
• Exact command to reproduce:

I am using TFX's evaluator

eval_config = tfma.EvalConfig(
  model_specs=model_specs,
  metrics_specs=tfma.metrics.specs_from_metrics([
      tf.keras.metrics.AUC(curve='ROC', name='ROCAUC'),
      tf.keras.metrics.AUC(curve='PR', name='PRAUC'),
      tf.keras.metrics.Precision(),
      tf.keras.metrics.Recall(),
      tf.keras.metrics.BinaryAccuracy(),
    ]),
  slicing_specs=slicing_specs
)

evaluator = Evaluator(
  eval_config=eval_config,
  model=model,
  examples=transform_examples,
)

context.run(evaluator)

Describe the problem

Running the same evaluation using Beam's DirectRunner locally will not cause any error, but whenever I run it on dataflow and when dataflow spawns more than one worker, I get an error like so:

output.with_value(self.phased_combine_fn.apply(output.value)): File "/usr/local/lib/python3.7/site-packages/apache_beam/transforms/combiners.py", line 882, in merge_only return self.combine_fn.merge_accumulators(accumulators) File "/home/sandbox/.pex/install/apache_beam-2.39.0-cp37-cp37m-linux_x86_64.whl.06f7ceb62380d1c704d774a5096a04f953de60c9/apache_beam-2.39.0-cp37-cp37m-linux_x86_64.whl/apache_beam/transforms/combiners.py", line 665, in merge_accumulators a in zip(self._combiners, zip(*accumulators_batch)) File "/home/sandbox/.pex/install/apache_beam-2.39.0-cp37-cp37m-linux_x86_64.whl.06f7ceb62380d1c704d774a5096a04f953de60c9/apache_beam-2.39.0-cp37-cp37m-linux_x86_64.whl/apache_beam/transforms/combiners.py", line 665, in a in zip(self._combiners, zip(*accumulators_batch)) File "/usr/local/lib/python3.7/site-packages/tensorflow_model_analysis/metrics/tf_metric_wrapper.py", line 560, in merge_accumulators for metric_index in range(len(self._metrics[output_name])): TypeError: 'NoneType' object is not subscriptable

Based on the dataflow log, the failing steps were:

• ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/CombineMetricsPerSlice/CombinePerKey(PreCombineFn)/Combine
• ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/CombineMetricsPerSlice/CombinePerKey(PreCombineFn)/GroupByKey
• ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/CombineMetricsPerSlice/CombinePerKey(PostCombineFn)/GroupByKey

I see that you have this commit, which appears to be addressing this problem, but it is immediately rolled back. I wonder if you have had similar issues and what would you recommend to fix the error.

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow Model Analysis): No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): GCP AI Vertex Workbench Debian 10
• TensorFlow Model Analysis installed from (source or binary): PyPI binary
• TensorFlow Model Analysis version (use command below): 0.26.0
• Python version: 3.7
• Jupyter Notebook version: n/a
• Exact command to reproduce:

You can obtain the TensorFlow Model Analysis version with

python -c "import tensorflow_model_analysis as tfma; print(tfma.version.VERSION)"

Describe the problem

I have a custom layer named MultiHeadAttention layer and when I ran the tfx pipeline, it shows a warning that it has a conflict with the default MultiHeadAttention layer and that I should rename the layer something else. When I rename it to CustomMultiHeadAttention, it suddenly breaks the tfx pipeline particularly in the evaluator component. When I don't change anything else in the code except reverting it back to the name "MultiHeadAttention" the evaluator component runs okay but the problem is that when trying to export the model/saving and loading, I'm also having some problems. What is the cause of this or is it a bug in tfma/tfx?

Source code / logs

Error when changing Custom Layer name from MultiHeadAttention -> CustomMultiHeadAttention

eval_config.py

import tensorflow_model_analysis as tfma

def set_eval_config() -> tfma.EvalConfig:

    eval_config = tfma.EvalConfig(
        model_specs=[
            tfma.ModelSpec(
                name="accent_model",
                signature_name="serving_evaluator",
                label_key="accent",
                prediction_key="accent_prediction",
            ),
            tfma.ModelSpec(
                name="phones_model",
                signature_name="serving_evaluator",
                label_key="target_phones",
                prediction_key="phone_predictions",
            ),
        ],
        metrics_specs=[
            tfma.MetricsSpec(
                output_names=["accent_prediction"],
                model_names=["accent_model"],
                metrics=[
                    tfma.MetricConfig(
                        class_name="AccentAccuracy",
                        module="aped.mlops.pipeline.metrics",
                    ),
                ],
            ),
            tfma.MetricsSpec(
                output_names=["phone_predictions"],
                model_names=["phones_model"],
                metrics=[
                    tfma.MetricConfig(
                        class_name="PhoneASRAccuracy",
                        module="aped.mlops.pipeline.metrics",
                        threshold=tfma.MetricThreshold(
                            value_threshold=tfma.GenericValueThreshold(lower_bound={"value": 0.01}),
                            change_threshold=tfma.GenericChangeThreshold(
                                direction=tfma.MetricDirection.HIGHER_IS_BETTER,
                                absolute={"value": -1e-10},
                            ),
                        ),
                    ),
                    tfma.MetricConfig(
                        class_name="PhoneErrorRate",
                        module="aped.mlops.pipeline.metrics",
                    ),
                    tfma.MetricConfig(
                        class_name="PhonesPrecision",
                        module="aped.mlops.pipeline.metrics",
                    ),
                    tfma.MetricConfig(
                        class_name="PhonesRecall",
                        module="aped.mlops.pipeline.metrics",
                    ),
                    tfma.MetricConfig(
                        class_name="PhonesF1Score",
                        module="aped.mlops.pipeline.metrics",
                    ),
                    tfma.MetricConfig(class_name="ExampleCount"),
                    tfma.MetricConfig(class_name="SparseCategoricalCrossentropy"),
                ],
            ),
        ],
        slicing_specs=[
            tfma.SlicingSpec(),
            tfma.SlicingSpec(feature_keys=["accent"]),
            tfma.SlicingSpec(feature_keys=["recording_length"]),
            tfma.SlicingSpec(feature_keys=["age"]),
            tfma.SlicingSpec(feature_keys=["gender"]),
            tfma.SlicingSpec(feature_keys=["bg_noise_type"]),
            tfma.SlicingSpec(feature_keys=["bg_noise_level"]),
            tfma.SlicingSpec(feature_keys=["english_level"]),
        ],
    )

    return eval_config

code snippet for evaluator component in tfx pipeline

evaluator = tfx.components.Evaluator(
        examples=transform.outputs["transformed_examples"],
        model=trainer.outputs["model"],
        # baseline_model=model_resolver.outputs['model'],
        eval_config=eval_config,
        example_splits=["eval"],
    )

multihead attention layer declaration snippet

class MultiHeadAttention(tf.keras.layers.Layer):
    """MultiHeadAttention Custom Layer"""

    def __init__(self, d_model: int, num_heads: int, dropout_rate: float, mixed_precision: bool = False) -> None:
        """Initialise the MultiHeadAttention Layer

        Args:
            d_model (int): Attention  modelling  dimension
            num_heads (int): Number of attention heads
            mixed_precision (bool, optional): True if the layer needs to handle mixed precision
            with float16. Defaults to False
        """
        super().__init__()
        self.num_heads = num_heads
        self.d_model = d_model
        self.dropout_rate = dropout_rate
        self.mixed_precision = mixed_precision

        assert d_model % self.num_heads == 0

        self.depth = d_model // self.num_heads

        init = tf.keras.initializers.RandomNormal(mean=0, stddev=np.sqrt(2.0 / (d_model + self.depth)))

        self.wq = tf.keras.layers.Dense(d_model, kernel_initializer=init)
        self.wk = tf.keras.layers.Dense(d_model, kernel_initializer=init)
        self.wv = tf.keras.layers.Dense(d_model, kernel_initializer=init)

        self.dense = tf.keras.layers.Dense(d_model, kernel_initializer="glorot_normal")