Parallelization of accelerated training tf data. Dataset generator

High quality resource sharing

Learning route guidance （ Click unlock ）	Knowledge orientation	Crowd positioning
🧡 Python Actual wechat ordering applet 🧡	Progressive class	This course is python flask+ Perfect combination of wechat applet , From the deployment of Tencent to the launch of the project , Create a full stack ordering system .
Python Quantitative trading practice	beginner	Take you hand in hand to create an easy to expand 、 More secure 、 More efficient quantitative trading system

When dealing with large-scale data , Data cannot be fully loaded into memory , We usually use two options

• Use tfrecords
• Use tf.data.Dataset.from_generator()

tfrecords Parallel use of The above has already been introduced , No more details here . If we don't want to generate tfrecord Intermediate document , Then the generator is what you need .

This paper mainly records that from_generator() Parallelization method of , stay tf.data in , Parallelization is mainly realized through map and num_parallel_calls Realization , But for some scenes , our generator() There is some processing logic in , It cannot be parallelized directly , The easiest way is to put generator() The logic in , Use map Realization .

tf.data.Dataset generator parallel

Yes generator() Complex logic in , We simplify it , That is, only some subscript value type operations are performed in the generator , take generator() In the processing section of py_function The parcel (wrapped) , And then call map Handle .

def func(i):
    i = i.numpy() # Decoding from the EagerTensor object
    x, y = your_processing_function(training_set[i])
    return x, y

z = list(range(len(training_set))) # The index generator

dataset = tf.data.Dataset.from_generator(lambda: z, tf.uint8)

dataset = dataset.map(lambda i: tf.py_function(func=func, 
                                               inp=[i], 
                                               Tout=[tf.uint8,
                                                     tf.float32]
                                               ), 
                      num_parallel_calls=tf.data.AUTOTUNE)

Because of implicit inference , Sometimes tensor Output shape It is unknown. , Need extra treatment

dataset = dataset.batch(8)
def \_fixup\_shape(x, y):
    x.set_shape([None, None, None, nb_channels]) # n, h, w, c
    y.set_shape([None, nb_classes]) # n, nb\_classes
    return x, y
dataset = dataset.map(_fixup_shape)

tf.Tensor And tf.EagerTensor

Why tf.py_function, Let's start with tf.Tensor And tf.EagerTensor

EagerTensor It's real time , You can get its value at any time , That is, through numpy obtain

Tensor It's not real time , It is a component in a static diagram , Only when feeding data 、 The... Can only be obtained after the operation is completed Tensor Value ,

map Function operation of mapping in , And just tell dataset, Every time you take out a sample, you should do it first function Used after an operation , therefore function Is called at each iteration dataset Is called when , Belong to Static diagram logic

tensorflow.python.framework.ops.EagerTensor
tensorflow.python.framework.ops.Tensor

tf.py_function What role does it play here ？

Wraps a python function into a TensorFlow op that executes it eagerly.

Just now map Data static diagram logic , The default parameters are Tensor. and Use tf.py_function() After packing , The parameter becomes EagerTensor.

references

【1】https://medium.com/@acordier/tf-data-dataset-generators-with-parallelization-the-easy-way-b5c5f7d2a18

【2】https://blog.csdn.net/qq_27825451/article/details/105247211

【3】https://www.tensorflow.org/guide/data_performance#parallelizing_data_extraction