Training addernet accelerated by CUDA
Usage
cd adder_cuda
python setup.py install
cd ..
python main.py
Environment
pytorch 1.10.0 CUDA 11.3
benchmark
| version | training_time_per_batch/s |
|---|---|
| raw | 1.61 |
| torch.cdist | 1.49 |
| cuda_unoptimized | 0.4508 |
| this work | 0.3158 |
The CUDA version of AdderNet has achieved a 5× speed increase over the original version. There seems to be some bugs in the Cuda_unoptimized version, causing the model to fail to converge. Its speed is still listed here for comparison. The experiment was run on RTX 2080Ti platform, and ResNet-20 based on CIFAR-10 was trained.
| Time(%) | Time | Calls | Avg | Min | Max | Name |
|---|---|---|---|---|---|---|
| 48.57 | 30.4752s | 3920 | 7.7743ms | 162.70us | 12.271ms | CONV_BACKWARD |
| 34.85 | 21.8686s | 19680 | 1.1112ms | 5.3770us | 11.827ms | _ZN2at6native27unrolled_elementwise_kernel... |
| 7.46 | 4.67901s | 5920 | 790.37us | 26.529us | 1.5841ms | CONV |
| 2.24 | 1.40372s | 3920 | 358.09us | 31.298us | 845.80us | col2im_kernel |
| 2.10 | 1.31882s | 36862 | 35.777us | 1.4720us | 276.24us | vectorized_elementwise_kernel |
| 1.43 | 900.03ms | 5920 | 152.03us | 7.9040us | 372.40us | im2col_kernel |
Here is the time distribution of training an epoch. If you are interested, you can continue to optimize the CUDA kernel.
6 Dec 13, 2022
6 Nov 26, 2022
56 Jan 02, 2023
2 May 05, 2022
75 Dec 09, 2022
72 Dec 23, 2022
87 Oct 10, 2022
22 Dec 29, 2022
88 May 15, 2022
9.3k Jan 07, 2023
425 Dec 22, 2022
16 Nov 21, 2022
856 Jan 06, 2023
76 Dec 08, 2022
77 Dec 26, 2022
312 Jan 08, 2023
1 Jan 18, 2022
35 Nov 14, 2022
2 Nov 23, 2021
50 Dec 16, 2022