[help] mindspire training based on ascend910 cannot reproduce the model effect on GPU

A very simple convolution down sampling model , Two layer convolution +ReLU

【 Training environment 】

1、Ascend910

2、MindSpore1.1.1

【 Problem description 】

Use the same training data and training process , stay Ascend Upper training convolution lower sampling model , Has been unable to reproduce in GPU The model effect trained on , Tests on the same test set MS-SSIM Yes 0.15 The gap between （0.945 VS 0.96）.

And a phenomenon was observed ：loss It's using MS-SSIM loss, stay GPU Used in training lr yes 0.00001, Finally, it can converge , But in Ascend910 In training , Or use it 0.00001 This lr, Train more than ten epoch after ,loss It will rise , Unable to converge . take lr To adjust to 0.0000001 Then training can converge , But the final model effect cannot be achieved GPU Model effect on .

Logically speaking , Just training such a simple model ,Ascend and GPU There should not be such a big gap , Now I have a few questions for help ：

1、 Use MindSpore In training , Are there any tuning trick？

2、 because loss It uses MindSpore Self contained Function mindspore.nn.MSSSIM Self realized loss, Is there a problem with this approach ？MindSpore Built in mindspore.nn.MSSSIM Implementation method and GPU Upper MSSSIM Is there any difference in the calculation method ？ Here is a custom MS-SSIM loss Realization ：

3、 Whether the difference in accuracy will affect the actual loss The effect of ？ such as ： stay fp16 Under precision ,MS-SSIM loss Will the effect of be reduced ？

4、 What are the possible reasons for Ascend Of MindSpore The training results of cannot be reproduced GPU（Pytorch） The result on ？

From your description above , If you are using fp16 Or mixing accuracy , Some operators may overflow , Suggestions for reference https://www.mindspore.cn/doc/api_python/zh-CN/r1.1/mindspore/mindspore.html?highlight=lossscale#mindspore.DynamicLossScaleManager  add to LossScale Or reference Enable automatic mixing accuracy — MindSpore r1.1 documentation  Use automatic mixing accuracy or manual mixing accuracy , It should be noted that if there is exp It is easy to overflow if you operate , You can check whether such an operator is used fp16, If so, it is suggested to use manual mixing and change the accuracy to fp32 Of