当前位置:网站首页>3、 How to customize data sets?
3、 How to customize data sets?
2022-07-29 06:07:00 【My hair is messy】
List of articles
Preface
MNIST Data, the most basic data set, has been destroyed by the students who are on the path of programming apes , So today I will teach you how to make a custom dataset .
One 、 Defined datasets , No pretreatment .
Here are some Inline code slice
.
import os
from torch.utils.data import Dataset ,DataLoader
from PIL import Image
#1. Create dataset class , Use torch.utils.data Medium Dataset Method .
class My_Dataset(Dataset):
#2. Loop to find the file path , And add labels
def __init__(self,main_dir,data_type,transforms):
self.dataset=[]# The empty list is the database with a new label
self.transforms=transforms
if data_type==0:
data_filename='train'
elif data_type is 1:
data_filename='val'
else:
data_filename='test'
for i , cls_filename in enumerate(
os.listdir(os.path.join(main_dir,data_filename))):
for i ,img_data in enumerate(os.listdir(
os.path.join(main_dir,data_filename,cls_filename))):
self.dataset.append([os.path.join(main_dir,
data_filename,cls_filename,img_data),int(img_data[0]) ])
#3. Calculate picture length , Convenient for later iterations
def __len__(self):
return len(self.dataset)# To get the length of the picture , Easy to iterate
#4、 Take out the image path , And open , Convenient for data preprocessing
def __getitem__(self, index):
img,label=self.dataset[index]
img_data=Image.open(img)
img_data=self.transforms(img_data)
return img_data,label
Example :@TOC
Two 、 Define datasets , And do data preprocessing .
Including rotation 、 tailoring 、 Into a tensor 、 expand 、 Regularization and so on .
1. Pretreatment part
#4、 Take out the image path , And open , Convenient for data preprocessing
def __getitem__(self, index):
img,label=self.dataset[index]
img_data=self.data_process(Image.open(img))
return img_data,label
#5. Data processing , Data to enhance 、 Add noise, etc
def data_process(self,x):
return transforms.Compose([transforms.ToTensor(),
transforms.Normalize(mean=(0.5,),std=(0.5,))])(x)
2. Define data process
The code is as follows ( Example ):
import os
from torchvision import transforms
from torch.utils.data import Dataset ,DataLoader
from PIL import Image
#1. Create dataset class , Use torch.utils.data Medium Dataset Method .
class My_Dataset(Dataset):
#2. Loop to find the file path , And add labels
def __init__(self,main_dir,data_type):
self.dataset=[]# The empty list is the database with a new label
if data_type==0:
data_filename='train'
elif data_type is 1:
data_filename='val'
else:
data_filename='test'
for i , cls_filename in enumerate(
os.listdir(os.path.join(main_dir,data_filename))):
for i ,img_data in enumerate(os.listdir(
os.path.join(main_dir,data_filename,cls_filename))):
self.dataset.append([os.path.join(main_dir,
data_filename,cls_filename,img_data),i ])
#3. Calculate picture length , Convenient for later iterations
def __len__(self):
return len(self.dataset)# To get the length of the picture , Easy to iterate
#4、 Take out the image path , And open , Convenient for data preprocessing
def __getitem__(self, index):
img,label=self.dataset[index]
img_data=self.data_process(Image.open(img))
return img_data,label
#5. Data processing , Data to enhance 、 Add noise, etc
def data_process(self,x):
return transforms.Compose([transforms.ToTensor(),
transforms.Normalize(mean=(0.5,),std=(0.5,))])(x)
``
边栏推荐
- The difference between asyncawait and promise
- fastText学习——文本分类
- The third week of postgraduate freshman training: resnet+resnext
- 迁移学习笔记——Adapting Component Analysis
- 【Attention】Visual Attention Network
- Improve quality with intelligence financial imaging platform solution
- Wechat built-in browser prohibits caching
- 研究生新生培训第一周:深度学习和pytorch基础
- 二、OCR训练时,将txt文件和图片数据转为lmdb文件格式
- Error in installing pyspider under Windows: Please specify --curl dir=/path/to/build/libcurl solution
猜你喜欢
ML11-SKlearn实现支持向量机
fastText学习——文本分类
Ribbon learning notes II
迁移学习——Transitive Transfer Learning
【语义分割】SETR_Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformer
Ribbon learning notes 1
"Full flash measurement" database acceleration solution
预训练语言模型的使用方法
【Transformer】AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
虚假新闻检测论文阅读(四):A novel self-learning semi-supervised deep learning network to detect fake news on...
随机推荐
迁移学习—— Transfer Feature Learning with Joint Distribution Adaptation
Power Bi report server custom authentication
第2周学习:卷积神经网络基础
Valuable blog and personal experience collection (continuous update)
[ml] PMML of machine learning model -- Overview
These process knowledge you must know
迁移学习—Geodesic Flow Kernel for Unsupervised Domain Adaptation
isAccessible()方法:使用反射技巧让你的性能提升数倍
Anr Optimization: cause oom crash and corresponding solutions
虚假新闻检测论文阅读(三):Semi-supervised Content-based Detection of Misinformation via Tensor Embeddings
[target detection] 6. SSD
Wechat applet source code acquisition (download with tools)
ML16-神经网络(2)
Improve quality with intelligence financial imaging platform solution
【目标检测】KL-Loss:Bounding Box Regression with Uncertainty for Accurate Object Detection
Research on the implementation principle of reentrantlock in concurrent programming learning notes
【Transformer】AdaViT: Adaptive Tokens for Efficient Vision Transformer
预训练语言模型的使用方法
ASM piling: after learning ASM tree API, you don't have to be afraid of hook anymore
关于Flow的原理解析