Mxnet record IO details

I am looking at the identification code recently , There are still many things worth learning , For example, distributed training , Data reading , Because the amount of training data identified is very large , Data production 、 Reading will greatly affect the training speed , Here is a summary of the data loading and reading section .

First , When we need to use large data sets to train models , At this time, loading and reading data becomes the bottleneck of our training . and Mxnet Provides a data storage format ImageRecord, With this data storage format, you can use multiple processes to load data , It can greatly improve the efficiency of training .
Mxnet Provides a mx.recordio modular , It can be done to ImageRecord File operation , This module contains two sub modules .MXRecordIO Support sequential read and write ,MXIndexedRecordIO Support random read and write . Let's take a look at , Some interface functions in these two sub modules , And how to use them to read and save ImageRecord file

1. Save the data as rec Format , And the corresponding index file
   

Save the picture as rec Format

import mxnet as mx

write_record = mx.recordio.MXIndexedRecordIO("test.idx","test.rec", 'w')
# Read the picture 
img_path = "img/test.jpg"
# Convert picture data to Numpy Array 
img = mx.image.imread(img_path).asnumpy()
label = 1.0
header = mx.recordio.IRHeader(flag=0,label=label,id=0,id2=0)
s = mx.recordio.pack_img(header,img,quality=95,img_fmt=".jpg")
# Writes data to rec In file 
write_record.write_idx(0,s)
write_record.close()

Read the picture rec Format

import mxnet as mx
read_record  = mx.recordio.MXIndexedRecordIO(path_imgidx, path_imgrec, 'r')
def __getitem__(self, index):
    idx = self.imgidx[index]
    s = self.imgrec.read_idx(idx)
    header, img = mx.recordio.unpack(s)
    label = header.label
    if not isinstance(label, numbers.Number):
        label = label[0]
    label = torch.tensor(label, dtype=torch.long)
    sample = mx.image.imdecode(img).asnumpy()

    ##########################################
    #sample = cv2.cvtColor(sample, cv2.COLOR_BGR2GRAY)
    #sample = cv2.cvtColor(sample, cv2.COLOR_GRAY2BGR)
    ##########################################

    if self.transform is not None:
        sample = self.transform(sample)
    return sample, label

here unpack Read it out MXImageRecord Decompress to image, label and img Change to the corresponding type and process transform, It can be used as the input of the network .

Reference resources ：
1.https://xiulian.blog.csdn.net/article/details/107210952
2. https://blog.csdn.net/qq_20622615/article/details/88633767