当前位置:网站首页>【torch】|torch. nn. utils. clip_ grad_ norm_
【torch】|torch. nn. utils. clip_ grad_ norm_
2022-07-06 05:18:00 【rrr2】
The greater the gradient ,total_norm The bigger the value is. , Leading to clip_coef The smaller the value of , Eventually, it will also lead to the more severe clipping of the gradient , Very reasonable.
norm_type Take... No matter how much , about total_norm The impact is not too great (1 and 2 The gap is a little larger ), So you can take the default value directly 2
norm_type The bigger it is ,total_norm The smaller it is ( The conclusions observed in the experiment , Math is not good , It will not prove that , So this article is not necessarily right )
...
loss = crit(...)
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=10, norm_type=2)
optimizer.step()
...
clip_coef The smaller it is , The more severe the cutting of gradient , namely , The more you reduce the value of the gradient
max_norm The smaller it is ,clip_coef The smaller it is , therefore ,max_norm The bigger it is , The softer the solution of gradient explosion ,max_norm The smaller it is , The harder to solve the gradient explosion .max_norm You can take decimals
ref
https://blog.csdn.net/Mikeyboi/article/details/119522689
边栏推荐
- Zoom and pan image in Photoshop 2022
- [buuctf.reverse] 159_[watevrCTF 2019]Watshell
- MySQL if and ifnull use
- 【LeetCode】18、四数之和
- C# AES对字符串进行加密
- The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
- Flody的应用
- Cuda11.1 online installation
- Nacos TC setup of highly available Seata (02)
- Postman test report
猜你喜欢
Excel转换为Lua的配置文件
Codeforces Round #804 (Div. 2)
TCP three handshakes you need to know
你需要知道的 TCP 三次握手
[mask requirements of OSPF and Isis in multi access network]
February 12 relativelayout
F12 solve the problem that web pages cannot be copied
Figure database ongdb release v-1.0.3
图论的扩展
Vite configures the development environment and production environment
随机推荐
C进阶-数据的存储(上)
图论的扩展
[lgr-109] Luogu may race II & windy round 6
Set detailed map + interview questions
nacos-高可用seata之TC搭建(02)
SQLite queries the maximum value and returns the whole row of data
集合详解之 Collection + 面试题
The ECU of 21 Audi q5l 45tfsi brushes is upgraded to master special adjustment, and the horsepower is safely and stably increased to 305 horsepower
Driver development - hellowdm driver
Idea one key guide package
Steady, 35K, byte business data analysis post
Mysql高级篇学习总结9:创建索引、删除索引、降序索引、隐藏索引
In 2022, we must enter the big factory as soon as possible
SQLite add index
MySQL advanced learning summary 9: create index, delete index, descending index, and hide index
Promotion hung up! The leader said it wasn't my poor skills
Modbus protocol communication exception
Lepton 无损压缩原理及性能分析
Postman assertion
C AES encrypts strings