当前位置:网站首页>Pytoch (VI) -- model tuning tricks
Pytoch (VI) -- model tuning tricks
2022-07-07 08:12:00 【CyrusMay】
Pytorch( 6、 ... and ) —— Model tuning tricks
1. Regularization Regularization
1.1 L1 Regularization
import torch
import torch.nn.functional as F
from torch import nn
device=torch.device("cuda:0")
MLP = nn.Sequential(nn.Linear(128,64),
nn.ReLU(inplace=True),
nn.Linear(64,32),
nn.ReLU(inplace=True),
nn.Linear(32,10)
)
MLP.to(device)
loss_classify = nn.CrossEntropyLoss().to(device)
# L1 norm
l1_loss = 0
for param in MLP.parameters():
l1_loss += torch.sum(torch.abs(param))
loss = loss_classify+l1_loss
1.2 L2 Regularization
import torch
import torch.nn.functional as F
from torch import nn
device=torch.device("cuda:0")
MLP = nn.Sequential(nn.Linear(128,64),
nn.ReLU(inplace=True),
nn.Linear(64,32),
nn.ReLU(inplace=True),
nn.Linear(32,10)
)
MLP.to(device)
# L2 norm
opt = torch.optim.SGD(MLP.parameters(),lr=0.001,weight_decay=0.1) # adopt weight_decay Realization L2
loss = nn.CrossEntropyLoss().to(device)
2 Momentum and learning rate decay
2.1 momentum
opt = torch.optim.SGD(model.parameters(),lr=0.001,momentum=0.78,weight_decay=0.1)
2.2 learning rate tunning
- torch.optim.lr_scheduler.ReduceLROnPlateau() Use when the value of the loss function does not decrease
- torch.optim.lr_scheduler.StepLR() Reduce the learning rate according to a certain number of steps
opt = torch.optim.SGD(net.parameters(),lr=1)
lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer=opt,mode="min",factor=0.1,patience=10)
for epoch in torch.arange(1000):
loss_val = train(...)
lr_scheduler.step(loss_val) # monitor loss
opt = torch.optim.SGD(net.parameters(),lr=1)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer=opt,step_size=30,gamma=0.1)
for epoch in torch.arange(1000):
lr_scheduler.step() # monitor loss
train(...)
3. Early Stopping
4. Dropout
model = nn.Sequential(
nn.Linear(256,128),
nn.Dropout(p=0.5),
nn.ReLu(),
)
by CyrusMay 2022 07 03
边栏推荐
- Network learning (II) -- Introduction to socket
- 复杂网络建模(二)
- 在Rainbond中实现数据库结构自动化升级
- 【无标题】
- [untitled]
- 拓维信息使用 Rainbond 的云原生落地实践
- Vulnerability recurrence fastjson deserialization
- 柯基数据通过Rainbond完成云原生改造,实现离线持续交付客户
- Find the mode in the binary search tree (use medium order traversal as an ordered array)
- 让Livelink初始Pose与动捕演员一致
猜你喜欢
JS cross browser parsing XML application
buureservewp(2)
央视太暖心了,手把手教你写HR最喜欢的简历
快解析内网穿透助力外贸管理行业应对多种挑战
Basic use of CTF web shrink template injection nmap
Myabtis_ Plus
Uniapp mobile terminal forced update function
The largest 3 same digits in the string of leetcode simple question
CDC (change data capture technology), a powerful tool for real-time database synchronization
[quick start of Digital IC Verification] 17. Basic grammar of SystemVerilog learning 4 (randomization)
随机推荐
Search for an element in a binary search tree (BST)
探索STEAM艺术设计中的创造力
CTF-WEB shrine模板注入nmap的基本使用
game攻防世界逆向
Complex network modeling (I)
[untitled]
【数字IC验证快速入门】14、SystemVerilog学习之基本语法1(数组、队列、结构体、枚举、字符串...内含实践练习)
2022 National latest fire-fighting facility operator (primary fire-fighting facility operator) simulation questions and answers
提高企业产品交付效率系列(1)—— 企业应用一键安装和升级
Complete linear regression manually based on pytoch framework
These five fishing artifacts are too hot! Programmer: I know, delete it quickly!
Fast parsing intranet penetration escorts the document encryption industry
【踩坑系列】uniapp之h5 跨域的问题
Blob 對象介紹
Myabtis_Plus
LeetCode简单题之判断一个数的数字计数是否等于数位的值
[quick start of Digital IC Verification] 17. Basic grammar of SystemVerilog learning 4 (randomization)
Rainbond 5.7.1 支持对接多家公有云和集群异常报警
C language queue
Recursive method to verify whether a tree is a binary search tree (BST)