我发现 PaddleViT 所有模型中的 README.md 都存在两个问题（以下均以 PaddleViT/image_classification/BEiT/ BEiT 模型的 README.md 为例）：

• 其一，Usage 示例代码中，加载预训练权重时少了后缀 .pdparams，而且注释中提到 .pdparams is NOT needed 也是不对的，应该是在下面的命令行参数中 -pretrained 的值是不需要 .pdparams，二者搞混了。

from config import get_config
from beit import build_beit as build_model
# config files in ./configs/
config = get_config('./configs/beit_base_patch16_224.yaml')
# build model
model = build_model(config)
# load pretrained weights, .pdparams is NOT needed
model_state_dict = paddle.load('./beit_base_patch16_224_ft22kto1k')
model.set_dict(model_state_dict)

应该讲注释注释中的 , .pdparams is NOT needed 删去，并在模型加载时，加上后缀 .pdparams：

from config import get_config
from beit import build_beit as build_model
# config files in ./configs/
config = get_config('./configs/beit_base_patch16_224.yaml')
# build model
model = build_model(config)
# load pretrained weights
model_state_dict = paddle.load('./beit_base_patch16_224_ft22kto1k.')
model.set_dict(model_state_dict)

• 其二，在 Evaluation 和 Training 的命令行参数值多加了一个单引号，如果在终端直接执行，会出现 FileNotFoundError 错误：

FileNotFoundError: [Errno 2] No such file or directory: "'./configs/beit_base_patch16_224.yaml'"

我之前在终端预训模型训练和验证的命令时，出现过这个错误，群里也有其他同学出现了这样的问题。出现这个错误的原因是因为 argparse 在解析命令行参数时，为字符串类型的参数值自动加上了一个双引号。所以，在为命令行参数赋值时，不需要加上引号。所以，应该去掉 Evaluation 和 Training 命令行参数值中的单引号。 单 GPU 验证：

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./beit_base_patch16_224_ft22kto1k'

我修改为：

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
    -cfg=./configs/beit_base_patch16_224.yaml \
    -dataset=imagenet2012 \
    -batch_size=16 \
    -data_path=/path/to/dataset/imagenet/val \
    -eval \
    -pretrained=/path/to/pretrained/model/beit_base_patch16_224_ft22kto1k  # .pdparams is NOT needed

多 GPU 验证：

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \
    -eval \
    -pretrained='./beit_base_patch16_224_ft22kto1k'

我修改为：

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg=./configs/beit_base_patch16_224.yaml \
    -dataset=imagenet2012 \
    -batch_size=16 \
    -data_path=/path/to/dataset/imagenet/val \
    -eval \
    -pretrained=/path/to/pretrained/model/beit_base_patch16_224_ft22kto1k  # .pdparams is NOT needed

单 GPU 训练：

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
  -cfg='./configs/beit_base_patch16_224.yaml' \
  -dataset='imagenet2012' \
  -batch_size=32 \
  -data_path='/dataset/imagenet' \

我修改为：

CUDA_VISIBLE_DEVICES=0 \
python main_single_gpu.py \
  -cfg=./configs/beit_base_patch16_224.yaml \
  -dataset=imagenet2012 \
  -batch_size=32 \
  -data_path=/path/to/dataset/imagenet/train \

多 GPU 训练：

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg='./configs/beit_base_patch16_224.yaml' \
    -dataset='imagenet2012' \
    -batch_size=16 \
    -data_path='/dataset/imagenet' \ 

我修改为：

CUDA_VISIBLE_DEVICES=0,1,2,3 \
python main_multi_gpu.py \
    -cfg=./configs/beit_base_patch16_224.yaml \
    -dataset=imagenet2012 \
    -batch_size=16 \
    -data_path=/path/to/dataset/imagenet/train \ 

一会儿，我再提交个 PR，请官方审查~

如果是多卡训练，则需要初始化多卡训练环境。

if nranks > 1:
    # Initialize parallel environment if not done.
    if not paddle.distributed.parallel.parallel_helper._is_parallel_ctx_initialized():
        logger.info("using dist training")
        # 初始化动态图模式下的并行训练环境，目前同时初始化NCCL和GLOO上下文用于通信。
        paddle.distributed.init_parallel_env()
        ddp_model = paddle.DataParallel(model)
    else:
        ddp_model = paddle.DataParallel(model)

不理解：if not paddle.distributed.parallel.parallel_helper._is_parallel_ctx_initialized(): 这个判断是什么意思，难道paddle.distributed.init_parallel_env()不应该是必须初始化的吗？

另外，还有如果要运行多机多卡需要修改什么代码吗？ 谢谢

Describe the bug resume training error AttributeError: 'Momentum' object has no attribute 'set_dict'

To Reproduce Steps to reproduce the behavior: 1.Go to 'PaddleViT/object_detection/Swin/' 2.Run 'python main_single_gpu.py -resume='./output/train-20211210-09-50-43/Swin-Epoch-45'

The recovery of model can pass

Screenshots Traceback (most recent call last): File "C:\Program Files\JetBrains\PyCharm Community Edition 2021.2.2\plugins\python-ce\helpers\pydev\pydevd.py", line 1483, in _exec pydev_imports.execfile(file, globals, locals) # execute the script File "C:\Program Files\JetBrains\PyCharm Community Edition 2021.2.2\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "F:/***/pp_swin/main_single_gpu.py", line 400, in <module> main() File "F:/***/pp_swin/main_single_gpu.py", line 313, in main optimizer.set_dict(opt_state) AttributeError: 'Momentum' object has no attribute 'set_dict'

Version (please complete the following information):

• Paddle Version: [ 2.2.0]
• Python Version [3.6]
• GPU/CPU mode [ Gpu]

Describe your feature request Will PaddleViT recently add Mobile-Former and release pretrained weights on ImageNet?

Describe the reference code or paper Paper -> Mobile-Former: Bridging MobileNet and Transformer

Describe the possible solution

Additional context Add any other context or screenshots about the feature request here.

Describe the bug PaddleViT-Seg的requirements.txt文档中需要安装的依赖库有如下：

• cityscapesScripts==2.2.0
• detail==4.0
• numpy==1.20.3
• opencv-python==4.5.2.52
• scipy==1.6.3
• yacs==0.1.8

问题一： 没有detail 这个库 自己解决方式：删除detail

问题二： opencv-python库没有4.5.2.52这个版本 自己解决方式：更换opencv-python版本

where can I get the cswin_large_224 pretrained 22k model and the 22kto1k labels mapping file. I want to fine-tune the cswin_large_224 model with imagenet-1k dataset myself.

When I am training the model, there is an error in calculating ACC. Could you provide a way for me to solve it?

Error message:Tensor holds the wrong type, it holds int, but desires to be int64_t.

pred and label type

I used the AIstudio GPU version and tried the DETR project.But got some CUDA mistake. I set the config. !pip install yacs import paddle from config import get_config from detr import build_detr config = get_config('./configs/detr_resnet50.yaml') model, critertion, postprocessors = build_detr(config) model_state_dict = paddle.load('detr_resnet50.pdparams') model.set_dict(model_state_dict) and this is the return W1216 20:21:43.708824 167 device_context.cc:404] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1 W1216 20:21:43.712436 167 device_context.cc:422] device: 0, cuDNN Version: 7.6. 100%|██████████| 151272/151272 [00:02<00:00, 69020.62it/s] And when I command sh run_eval.sh I got some Warning and Error. W1216 20:22:31.344815 398 init.cc:141] Compiled with WITH_GPU, but no GPU found in runtime. /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/framework.py:301: UserWarning: You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default. "You are using GPU version Paddle, but your CUDA device is not set properly. CPU device will be used by default."

Traceback (most recent call last): File "main_single_gpu.py", line 321, in <module> main() File "main_single_gpu.py", line 174, in main paddle.seed(seed) File "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/framework/random.py", line 46, in seed for i in range(core.get_cuda_device_count()): OSError: (External) Cuda error(100), no CUDA-capable device is detected. [Advise: Please search for the error code(100) on website( https://docs.nvidia.com/cuda/archive/10.0/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038 ) to get Nvidia's official solution about CUDA Error.] (at /paddle/paddle/fluid/platform/gpu_info.cc:99)

训练DETR遇到的报错，出现的时间不固定，有时候是在训练刚开始几个batch时，有时候是训练了几个epoch后会出现。单卡3090，显存占用率基本维持在50%左右

训练使用的命令： CUDA_VISIBLE_DEVICES=1
python main_single_gpu.py
-cfg='./configs/detr_resnet50.yaml'
-dataset='coco'
-batch_size=2
-data_path='/dataset/coco' \

报错： Traceback (most recent call last): File "main_single_gpu.py", line 321, in main() File "main_single_gpu.py", line 289, in main accum_iter=config.TRAIN.ACCUM_ITER) File "main_single_gpu.py", line 91, in train loss_dict = criterion(outputs, targets) File "/home/cuiyuan/anaconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 902, in call outputs = self.forward(*inputs, **kwargs) File "/disk/disk1/quyi/PaddleViT/object_detection/DETR/detr.py", line 285, in forward indices = self.matcher(outputs_without_aux, targets) # list of index(tensor) pairs File "/home/cuiyuan/anaconda3/envs/paddle2/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py", line 902, in call outputs = self.forward(*inputs, **kwargs) File "/disk/disk1/quyi/PaddleViT/object_detection/DETR/matcher.py", line 129, in forward idx = linear_sum_assignment(c[i]) File "/home/cuiyuan/anaconda3/envs/paddle2/lib/python3.7/site-packages/scipy/optimize/_lsap.py", line 100, in linear_sum_assignment return _lsap_module.calculate_assignment(cost_matrix) ValueError: matrix contains invalid numeric entries

在 vit transformer 的实现中（ViT Transformer Attention），多头注意力的 attn_head_size 的计算是由传入的 embed_dim 和 num_heads 计算得到的：

self.attn_head_size = int(embed_dim / self.num_heads)

我认为这里的实现至少有两个问题：

• 其一，没有对embed_dim是否能num_heads整除做检查。当embed_dim不能被num_heads整除，或者num_heads > embed_dim时，transpose_multihead 的操作会出现异常：

    def transpose_multihead(self, x):
        new_shape = x.shape[:-1] + [self.num_heads, self.attn_head_size]
        x = x.reshape(new_shape)
        x = x.transpose([0, 2, 1, 3])
        return x

• 其二，attn_head_size 的大小受到 embed_dim 和 num_heads 的限制，当预训练模型时，不能随意设置 attn_head_size 的大小，代码不够灵活。

解决上述问题的办法，就是为 Attention 的 __init__ 方法添加一个 attn_head_size 的参数，这样即不影响现有预训练模型的加载，又可以在预训练时，灵活设置 attn_head_size 的大小。由于 attn_head_size 与输入维度 embed_dim 无关，也不需要验证 embed_dim 是否能被 num_heads 整除。 目前主流框架中，两种实现都有： 第一种，由 embed_dim 和 num_heads 参数计算 attn_head_size 的实现，包括: PaddlePaddle: https://github.com/PaddlePaddle/Paddle/blob/develop/python/paddle/nn/layer/transformer.py#L109 PyTorch: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/transformer.py transformers: https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L226 第二种，将 attn_head_size 作为参数传入的实现，包括： TensorFlow: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/layers/multi_head_attention.py#L126 TensorFlow Addons: https://github.com/tensorflow/addons/blob/master/tensorflow_addons/layers/multihead_attention.py 我个人非常推荐第二种实现方式，API 使用起来更加灵活，代码看起来也非常顺畅，更加合理。 比如，原实现中 all_head_size 的定义：

self.all_head_size = self.attn_head_size * self.num_heads

all_head_size == embed_dim，完全没有必要定义。这个变量，只在 __init__ ：

        self.qkv = nn.Linear(embed_dim,
                             self.all_head_size*3,  # weights for q, k, and v
                             weight_attr=w_attr_1,
                             bias_attr=b_attr_1 if qkv_bias else False)

和 forward ：

new_shape = z.shape[:-2] + [self.all_head_size]

中用到。__init__ 中的 qkv 映射的输出维度 self.all_head_size*3 可改为 embed_dim*3，forward中的 new_shape 用到的 self.all_head_size，可以在方法的开始，取出输入 x 的维度，修改如下:

embed_dim = x.shape[-1]
……
new_shape = z.shape[:-2] + [embed_dim]

以上是我对源码中定义 self.all_head_size 的质疑。 还有最后输出加一层 Linear Layer 的必要性：

        self.out = nn.Linear(embed_dim,
                             embed_dim,
                             weight_attr=w_attr_2,
                             bias_attr=b_attr_2)

在 forward 中，最后输出执行线性映射操作的上面由一行注释 reshape，

        z = z.reshape(new_shape)
        # reshape
        z = self.out(z)

意思应该是将维度映射回输入维度 embed_dim，方面后面的残差连接。不过既然 all_head_size == embed_dim，那何来 reshape? 所以，我认为这里对输出的线性映射是不必要的。 不过，如果我们使用第二种方式实现，将 attn_head_size 作为参数传入，不依赖 embed_size 和 num_heads 来计算，以上代码看起来就顺畅多了，合理多了。 第二种实现，将 attn_head_size 作为参数传入，只需在源代码基础上更改几行代码即可，实现如下：

from typing import Tuple, Union

import paddle
import paddle.nn as nn
from paddle import ParamAttr
from paddle import Tensor


class Attention(nn.Layer):
    """ Attention module

    Attention module for ViT, here q, k, v are assumed the same.
    The qkv mappings are stored as one single param.

    Attributes:
        num_heads: number of heads
        attn_head_size: feature dim of single head
        all_head_size: feature dim of all heads
        qkv: a nn.Linear for q, k, v mapping
        scales: 1 / sqrt(single_head_feature_dim)
        out: projection of multi-head attention
        attn_dropout: dropout for attention
        proj_dropout: final dropout before output
        softmax: softmax op for attention
    """
    def __init__(self,
                 embed_dim: int,
                 num_heads: int,
                 attn_head_size: int,
                 qkv_bias: Union[bool, ParamAttr],
                 dropout: float = 0.,
                 attention_dropout: float = 0.):
        super().__init__()
        """
        增加了一个attn_head_size的参数，attn_head_size和num_heads的大小不受embed_dim的限制，使API的使用更灵活。
        """
        self.num_heads = num_heads
        # self.attn_head_size = int(embed_dim / self.num_heads)
        self.attn_head_size = attn_head_size
        self.all_head_size = self.attn_head_size * self.num_heads  # Attention Layer's hidden_size

        w_attr_1, b_attr_1 = self._init_weights()
        self.qkv = nn.Linear(embed_dim,
                             self.all_head_size*3,  # weights for q, k, and v
                             weight_attr=w_attr_1,
                             bias_attr=b_attr_1 if qkv_bias else False)

        self.scales = self.attn_head_size ** -0.5

        w_attr_2, b_attr_2 = self._init_weights()
        # self.out = nn.Linear(embed_dim,
        #                      embed_dim,
        #                      weight_attr=w_attr_2,
        #                      bias_attr=b_attr_2)
        # 汇总多头注意力信息，并将维度映射回输入维度embed_dim，方便残差连接
        self.out = nn.Linear(self.all_head_size,
                             embed_dim,
                             weight_attr=w_attr_2,
                             bias_attr=b_attr_2)

        self.attn_dropout = nn.Dropout(attention_dropout)
        self.proj_dropout = nn.Dropout(dropout)
        self.softmax = nn.Softmax(axis=-1)

    def _init_weights(self) -> Tuple[ParamAttr, ParamAttr]:
        weight_attr = paddle.ParamAttr(initializer=nn.initializer.KaimingUniform())
        bias_attr = paddle.ParamAttr(initializer=nn.initializer.KaimingUniform())
        return weight_attr, bias_attr

    def transpose_multihead(self, x: Tensor) -> Tensor:
        new_shape = x.shape[:-1] + [self.num_heads, self.attn_head_size]
        x = x.reshape(new_shape)
        x = x.transpose([0, 2, 1, 3])
        return x

    def forward(self, x: Tensor) -> Tuple[Tensor, Tensor]:
        qkv = self.qkv(x).chunk(3, axis=-1)
        q, k, v = map(self.transpose_multihead, qkv)

        attn = paddle.matmul(q, k, transpose_y=True)
        attn = attn * self.scales
        attn = self.softmax(attn)
        attn_weights = attn
        attn = self.attn_dropout(attn)

        z = paddle.matmul(attn, v)
        z = z.transpose([0, 2, 1, 3])
        new_shape = z.shape[:-2] + [self.all_head_size]
        z = z.reshape(new_shape)
        # 汇总多头注意力信息，并将维度映射回输入维度embed_dim，方便残差连接
        z = self.out(z)
        z = self.proj_dropout(z)
        return z, attn_weights

测试：

def main():
    t = paddle.randn([4, 16, 96])     # [batch_size, num_patches, embed_dim]
    print('input shape = ', t.shape)

    model = Attention(embed_dim=96,
                      num_heads=8,
                      attn_head_size=128,
                      qkv_bias=False,
                      dropout=0.,
                      attention_dropout=0.)

    print(model)

    out, attn_weights = model(t)
    print(out.shape)
    print(attn_weights.shape)

    for name, param in model.named_parameters():
        print(f'param name: {name},\tparam shape: {param.shape} ')


if __name__ == "__main__":
    main()

输出：

input shape =  [4, 16, 96]
Attention(
  (qkv): Linear(in_features=96, out_features=3072, dtype=float32)
  (out): Linear(in_features=1024, out_features=96, dtype=float32)
  (attn_dropout): Dropout(p=0.0, axis=None, mode=upscale_in_train)
  (proj_dropout): Dropout(p=0.0, axis=None, mode=upscale_in_train)
  (softmax): Softmax(axis=-1)
)
[4, 16, 96]
[4, 8, 16, 16]
param name: qkv.weight,	param shape: [96, 3072] 
param name: out.weight,	param shape: [1024, 96] 
param name: out.bias,	param shape: [96] 

以上是我个人的一点儿不成熟的小建议，望官方评估采纳~

在PaddleViT用于语义分割的Trans2Seg模型，配置文件里需要使用resnet50c.pdparams的预训练权重文件，想问下是否可以提供一下下载方式？如果有resnet50, resnet101，resnet50c, resnet101c等系列模型文件能够提供一份吗？

感谢！@xperzy

Describe your feature request Any chance we could get the pretrained ViT model with MAE?

Describe the reference code or paper Masked Autoencoders Are Scalable Vision Learners

Describe the possible solution Please pretrain the ViT with MAE and upload the weight link to the readme file. Thanks in advance!

Additional context None.

Describe your feature request 您好，请问MobileFormer关于目标检测方面的复现有吗？

Describe the reference code or paper

Describe the possible solution

Additional context Add any other context or screenshots about the feature request here.

我想计算基于Transformer的一些分割模型的FLOPs用于评估实验模型的性能, 使用的是Paddle.flops来计算，但是出现了一些无法统计的情况（ Treat it as zero FLOPs），是不是这样计算出来的FLOPs的值是不准确的呢？可以如何解决？

我看到的是需要自己定义用于实现对自定义网络层的统计，也就是参数custom_ops，目前只找到了官方提供的用于计算paddle.nn.SyncBatchNorm的方法， 但是我不知道怎么设计用于计算.MaxPool2D、.LayerNorm、.GroupNorm、 .GELU、 Embedding这些函数，如果不计算的话，最后得出的FLOPs值应该有较大的差异，可以帮我解决这个问题吗？

下面是的的结果：

...