Reproduction process and problems of analog transformer (ICLR 2022 Spotlight)

Official code ：GitHub - thuml/Anomaly-Transformer: About Code release for "Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight), https://openreview.net/forum?id=LzQQ89U1qm_

The author recommends python3.6,pytorch 1.4 

1. Environmental changes

Try installing  pytorch 1.4 function , But the code will get stuck , And there's no mistake . The positioning error is ：Anomaly-Transformer/model/attn.py

self.distances = torch.zeros((window_size, window_size)).cuda()

.cuda() Get stuck ： as a result of Installed  pytorch 1.4 Corresponding CUDA Version is 10.x, The power of calculation is sm_86,CUDA 10.x Highest support to sm_75, Therefore need CUDA 11.x To support sm_8.x.

---

So upgrade My environment python3.7,  pytorch 1.12  , The graphics card 3080Ti, CUDA edition ：11.3

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Run the training script again , An error again ：

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [512, 25]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

solve ： Comment out Anomaly-Transformer/solver.py One of the first .step()：

                # Minimax strategy
                loss1.backward(retain_graph=True)
                # self.optimizer.step()
                loss2.backward()
                self.optimizer.step()

Reference resources ：Why the optimizer.step() write twice？ · Issue #8 · thuml/Anomaly-Transformer · GitHub

---

2.   Congratulations ！  Running successfully ！

python main.py --anormly_ratio 1 --num_epochs 3    --batch_size 128  --mode train --dataset PSM  --data_path dataset/PSM --input_c 25    --output_c 25

------------ Options -------------
anormly_ratio: 1.0
batch_size: 128
data_path: dataset/PSM
dataset: PSM
input_c: 25
k: 3
lr: 0.0001
mode: train
model_save_path: checkpoints
num_epochs: 3
output_c: 25
pretrained_model: None
win_size: 100 

======================TEST MODE======================
/opt/conda/lib/python3.7/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
  warnings.warn(warning.format(ret))
Threshold : 0.002150955616962149
pred:    (87800,)
gt:      (87800,)
pred:  (87800,)
gt:    (87800,)
Accuracy : 0.9848, Precision : 0.9713, Recall : 0.9739, F-score : 0.9726 

The results in the paper ： about PSM Data sets

P: 96.91,R: 98.9,  F1: 97.89

Recurring Recall Slightly lower . however Precision Slightly higher . The two need to be weighed . You can adjust the above  Threshold : 0.002150955616962149 Balance the two .