Mindspire series one loading image classification data set

        MindSpore It provides loading interfaces for most common datasets and standard format datasets , You can use it directly mindspore.dataset Data loading is performed by the corresponding dataset loading class in , Such as MNIST、CIFAR-10、CIFAR-100、VOC、COCO、ImageNet、CelebA、CLUE etc. , And industry standard data sets , Include MindRecord、TFRecord、Manifest etc. .

Common datasets loaded to cifar10 For example , First of all, will cifar10 Data sets are downloaded and extracted locally .

1、 load cifar10 Data sets ：

DATA_DIR = "./cifar-10-batches-bin/"
sampler = ds.SequentialSampler(num_samples=5)
dataset = ds.Cifar10Dataset(DATA_DIR, sampler=sampler)

use create_dict_iterator Create data iterators to access data ：

for data in dataset.create_dict_iterator():
    print("Image shape: {}".format(data['image'].shape), ", Label: {}".format(data['label']))

2、 Load custom image classification dataset

         Use mindspore Load custom image classification data , have access to mindspore.dataset.ImageFolderDataset Interface to load . Put images of the same category in the same folder , Different categories are distinguished by different folders , Pass in the parent directory of all categories ImageFolderDataset Interface ,mindspore The image data will be automatically loaded and the corresponding labels will be assigned according to different folders .

 

          Here we use TinyImageNet As an example to load data . First , Use imageFolderDataset Interface incoming data path , adopt num_parallel_worker The number of data loading parallel threads can be set ,shuffle Whether the parameter setting disturbs the data order . In addition, we need to pass map Interface for image data preprocessing , Image preprocessing interface mindspore.dataset.vision.c_transforms, adopt c_transforms Image decoding is possible , Scaling normalization , Matrix transpose and other operations .

import mindspore
import mindspore.dataset as ds
import mindspore.dataset.vision.c_transforms as CV
import mindspore.dataset.transforms.c_transforms as C
from mindspore import dtype as mstype

def create_dataset(data_path, batch_size=24, c_transforms
 repeat_num=1):
    """ Define datasets """
    parallel_mode = context.get_auto_parallel_context("parallel_mode")
    if parallel_mode == context.ParallelMode.DATA_PARALLEL:
        data_set = ds.ImageFolderDataset(data_path, num_parallel_def create_dataset(data_path, batch_size=24, repeat_num=1):
    """ Define datasets """
    data_set = ds.ImageFolderDataset(data_path, num_parallel_workers=8, shuffle=True)
    image_size = [100, 100]
    mean = [0.485 * 255, 0.456 * 255, 0.406 * 255]
    std = [0.229 * 255, 0.224 * 255, 0.225 * 255]
    trans = [
        CV.Decode(),
        CV.Resize(image_size),
        CV.Normalize(mean=mean, std=std),
        CV.HWC2CHW()
    ]

    #  To achieve data map mapping 、 Batch processing and data repetition 
    type_cast_op = C.TypeCast(mstype.int32)
    data_set = data_set.map(operations=trans, input_columns="image", num_parallel_workers=8)
    data_set = data_set.map(operations=type_cast_op, input_columns="label", num_parallel_workers=8)
    data_set = data_set.batch(batch_size, drop_remainder=True)
    data_set = data_set.repeat(repeat_num)

    return data_set

         Data iteration ,ImageFolderDataset adopt create_tuple_iterator() Interface iterates over the data set , One at a time batch The data of .

if __name__ == '__main__':
    datapath = 'D:/Sources/Data/datasets/TinyImageNet/val'
    ds = create_dataset(datapath, batch_size=8)
    iterator = ds.create_tuple_iterator()
    for item in iterator:
        print(f'images:{mindspore.Tensor(item[0]).shape},labels:{item[1]}')