Mix sampling and loading with mindspire multiple datasets

Problem description ：

【 Function module 】

Mindspore 1.5

【 Operation steps & Problem phenomenon 】

1、 Positive and negative examples need to be loaded in comparative learning , From different data sets  PosDataSet as well as NegDataSet, also The sample size of the two is inconsistent , Now, it is used for two data sets respectively RandomSampler sampling

     Use batch The data sets are processed as PosBatchDataSet as well as NegBatchDataSet

2、 In order to use Model Multi card parallel function , Now you need to merge the two data sets into the same data set , When primary PosBatchDataSet perhaps NegBatchDataSet Of batchSize by 64 when , You want to merge the data set batchSize by 128, The top 64 The data comes from PosBatchDataSet, after 64 individual originate NegBatchDataSet

3. problem ：

    a. Whether it is feasible to merge data sets , If possible, can you provide reference

    b. Whether it can be used DistributedSampler Using multiple machines

    c.   because PosDataSet as well as NegDataSet Sample size is inconsistent , Can you specify NegDataSet Repeat sampling , until PosDataSet End of non repeated sampling

【 Screenshot information 】

class MakeDataset:

    def __init__(self, length):

        self.length = length

        self.A = [np.ones((2, 2)) for _ in range(length)]

        self.B = np.arange(length)

    def __getitem__(self, index):

        return self.A[index], self.B[index]

    def __len__(self):

        return self.length

    

batch_size = 64

pos_data_set = MakeDataset(1000)

neg_data_set = MakeDataset(1000)

pos_data_set = GeneratorDataset(pos_data_set, column_names=["A", "B"])

neg_data_set = GeneratorDataset(neg_data_set, column_names=["A", "B"])

pos_batch_data_set = pos_data_set.batch(batch_size)

neg_batch_data_set = neg_data_set.batch(batch_size)

# Merge(pos_batch_data_set, neg_batch_data_set)

answer ：

Make a proposal to NegBatchData and PosBatchData Put it in the same MakeDataset Inside , And then control it NegSample -> NegBatch and PosSample -> PosBatch The logic of , And repeat sampling can also be done in __getitem__ Internal control implementation . namely ： adopt MakeDataset The output is already mixed Batch. According to the above logic , only one MakeDataset, Yes, you can use DistributedSampler Of .