Is there any way to see the attention visualization for Bert Questioning and Answering model ?? Because I couldn't see BertForQuestionAnswering class in bertviz.pytorch_transformers_attn? I have fine-tuned over a QA dataset using hugging-face transformers and wanted to see the visualization for it. Can you suggest any way of doing it ??

It seems you removed encode_plus, what is the successor? All the notebook includes inputs = tokenizer.encode_plus(text, return_tensors='pt', add_special_tokens=True) which is wrong and raise an error.

Hi, Thank you for this great work. can I use this code to plot my model(I am useing BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)

model_type = 'bert' model_version = 'bert-base-uncased' do_lower_case = True model = model #(this my model) #tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True) tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case) sentence_a = sentences[0] sentence_b = sentences[1] call_html() show(model, model_type, tokenizer, sentence_a, sentence_b) I changed only the model with my model, and the sentences and I got this error??!!please help or share any blog that explain how to plot my model AttributeError: 'BertTokenizer' object has no attribute 'cls_token'

Thank you in advance

Is there any way to use bertviz to visualise the importance of the different words respect to a given prediction of a classification task (BertClassifier)? Similar to this: https://docs.fast.ai/text.interpret.html#interpret

Thank you

I'm using colab but it doesn't work. Help.

%%javascript require.config({ paths: { d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min', jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min', } });

def` show_head_view(model, tokenizer, sentence_a, sentence_b=None):

inputs = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', 

add_special_tokens=True)

input_ids = inputs['input_ids']
if sentence_b:
    token_type_ids = inputs['token_type_ids']
    attention = model(input_ids, token_type_ids=token_type_ids)[-1]
    sentence_b_start = token_type_ids[0].tolist().index(1)
else:
    attention = model(input_ids)[-1]
    sentence_b_start = None
input_id_list = input_ids[0].tolist() # Batch index 0
tokens = tokenizer.convert_ids_to_tokens(input_id_list)    
head_view(attention, tokens, sentence_b_start)

model_version = 'bert-base-uncased' do_lower_case = True

model = BertModel.from_pretrained(model_version, output_attentions=True) tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)

sentence_a = "the cat sat on the mat" sentence_b = "the cat lay on the rug"

show_head_view(model, tokenizer, sentence_a, sentence_b)

Hi,

Thank you for this fascinating work.

I tried to visualize T5 attentions on a high-Ram Colab Notebook with TPU. It runs perfectly when the input is short. However, when the input length is more than a few sentences, Colab notebook seems to crash. It's required in my research project that at most several paragraphs be visualized. Do you know if there is a way to make this work?

Thank you! Yifu (Charles)

Hello,

We appreciate your assistance with this helpful visualization for Bert. This issue occurs when I use the Asafaya pretrained model for the Arabic language, but not when I use the bert-base-multilingual-cased model.

Any suggestions!

best,

Hi, first of all, thank you for a great tool.

My question is how to visualize attention for translation? I would like to see how much particular input word attended the choice of output word. I am using Seq2Seq model and its output has three types od attention values: encoder, decoder and cross attentions. https://huggingface.co/transformers/main_classes/output.html#seq2seqlmoutput

So to plot it, It would mean that words on the right side of head_view come from the translated sequence.

Is it how seq2seq attention can be visualised?

Thank you even for directing me in advance. Below is a code I used to get Seq2SeqLMOutput

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from bertviz import model_view

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-pl-en")
model = AutoModelForSeq2SeqLM.from_pretrained("Helsinki-NLP/opus-mt-pl-en")

text = "Kot nie przekroczył drogi bo była za szeroka"

inputs = tokenizer.encode(text, return_tensors="pt")
outputs = model.generate(inputs)

tokens = tokenizer.convert_ids_to_tokens(inputs[0])
output = model(inputs, decoder_input_ids=inputs, output_attentions=True)

BertModel finetuned for a sequence classification task does not give expected results on visualisation. Ideally, the pretrained model should be loaded into BertForSequenceClassification, but that model does not return attentions scores for visualisation. When loaded into BertModel (0 to 11 layers), I assume the 11th layer (right before classification layer in BertForSequenceClassification) is the right layer to check attention distribution. But every word is equally attentive to every other word. I am wondering what can be the possible reasons and how I can fix it. Thanks.

Hi,

I'm trying to visualize the attention on my fine-tuned model, while the demo case works fine with us but when I try to feed attention results output by my fine-tuned model it's just showing blank but no exception was caught. I've been tried to slice the attention shape to the same as the one in demo case with [1,12,8,8] but still not working.

Hi, I have trained an XLM model that translates from English to Spanish. A model for this language pair is not available on huggingface's repo. Is there any way to load my saved model?

I didn't see any figure when running the code below. Is there something that I missed? Help me, pls.

from transformers import AutoTokenizer, AutoModel, utils
from bertviz import model_view

utils.logging.set_verbosity_error()  # Remove line to see warnings

# Initialize tokenizer and model. Be sure to set output_attentions=True.
# Load BART fine-tuned for summarization on CNN/Daily Mail dataset
model_name = "facebook/bart-large-cnn"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, output_attentions=True)

# get encoded input vectors
encoder_input_ids = tokenizer(utterances, return_tensors="pt", add_special_tokens=True).input_ids

# create ids of encoded input vectors
decoder_input_ids = tokenizer("Jane made a 9 PM reservation for 6 people tonight at Vegano Resto .", return_tensors="pt", add_special_tokens=True).input_ids

outputs = model(input_ids=encoder_input_ids, decoder_input_ids=decoder_input_ids)

encoder_text = tokenizer.convert_ids_to_tokens(encoder_input_ids[0])
decoder_text = tokenizer.convert_ids_to_tokens(decoder_input_ids[0])

model_view(
    encoder_attention=outputs.encoder_attentions,
    decoder_attention=outputs.decoder_attentions,
    cross_attention=outputs.cross_attentions,
    encoder_tokens= encoder_text,
    decoder_tokens=decoder_text
)

Hi! During looking through issues I found out that previously head_view_bart.ipynb example was existing in this repo, but now it only can be found through history in deleted branch: https://github.com/jessevig/bertviz/blob/b088f44dd169957dbe89019b81243ef5cf5e9dcb/notebooks/head_view_bart.ipynb

Could you tell us why this example (along with many others) was removed? It works perfectly fine now

Hi, thanks for the great visualization tool!

I'm just wondering whether we can have a feature which renders head view in horizontal direction? The reason is that it's more suitable to show the sequence of tokens in the horizontal direction for language like Chinese, Japanese or Korean.

In the above example, typical sentences in Chinese take about 6,70 characters but it already uses a lot of space showing 10 of them in the current head view.

Thanks again for the great tool!

Thanks for the great tool!

It would be nice to be able to save the visualizations for specific layers/heads as images. I have not been able to find a spot in the model/head/neuron_view.js file to add a saving function.

Do you maybe have a suggestion on how to save the visualizations as images?

Thanks!