当前位置：网站首页>Tensorflow [actual Google deep learning framework] uses HDF5 to process large data sets with tflearn

Tensorflow [actual Google deep learning framework] uses HDF5 to process large data sets with tflearn

2022-06-11 22:39:00 【Li Xiang superb】

List of articles

1.HDF5 file
2.github Code ：
3. Source code address ：
4.tf Introduction collection

1.HDF5 file

Tensorflow【 actual combat Google Deep learning framework 】 Use HDF5 Dealing with large datasets with TFLearn_ Data sets up to now , All the datasets we use can be loaded into memory . For small data sets , We can load all the image data into memory , Pre treatment , And forward propagation processing . However , For large data sets ( such as ImageNet), We need to create a data generator , Only a small number of datasets are accessed at a time （ such as mini-batch）, Then on batch Data preprocessing and forward propagation .

Keras Module is very convenient for data loading , You can use the original file path on disk as input to the training process . You don't need to store the entire data set in memory —— Just for Keras The data generator provides the image path , The generator automatically loads data from the path and propagates forward .

However , This method is very inefficient . It takes one to read every image on disk I/O operation , This will cause some delay . Training deep learning networks themselves is slow enough , So we should try to avoid I/O bottleneck .

A more reasonable solution is to generate the original image HDF5 Data sets ,, It's just that this time we're storing the original image , Instead of extracted features .HDF5 Not only can you store a large number of data sets , And it can also be used for I/O operation , Especially for extracting... From files batch( be called “ slice ”). We save the original image on disk to HDF5 In file , This allows the model to quickly traverse the dataset and train the deep learning network on it .

2.github Code ：

       
        # -*- coding: utf-8 -*-
        
"""
        
Example on how to use HDF5 dataset with TFLearn. HDF5 is a data model,
        
library, and file format for storing and managing data. It can handle large
        
dataset that could not fit totally in ram memory. Note that this example
        
just give a quick compatibility demonstration. In practice, there is no so
        
real need to use HDF5 for small dataset such as CIFAR-10.
        
"""
        

        
from __future__ import division, print_function, absolute_import
        

        
import tflearn
        
from tflearn.layers.core import *
        
from tflearn.layers.conv import *
        
from tflearn.data_utils import *
        
from tflearn.layers.normalization import *
        
from tflearn.layers.estimator import regression
        

        
# CIFAR-10 Dataset
        
from tflearn.datasets import cifar10
        
(X, Y), (X_test, Y_test) = cifar10.load_data()
        
Y = to_categorical(Y)
        
Y_test = to_categorical(Y_test)
        

        
# Create a hdf5 dataset from CIFAR-10 numpy array
        
import h5py
        
h5f = h5py.File('data.h5', 'w')
        
h5f.create_dataset('cifar10_X', data=X)
        
h5f.create_dataset('cifar10_Y', data=Y)
        
h5f.create_dataset('cifar10_X_test', data=X_test)
        
h5f.create_dataset('cifar10_Y_test', data=Y_test)
        
h5f.close()
        

        
# Load hdf5 dataset
        
h5f = h5py.File('data.h5', 'r')
        
X = h5f['cifar10_X']
        
Y = h5f['cifar10_Y']
        
X_test = h5f['cifar10_X_test']
        
Y_test = h5f['cifar10_Y_test']
        

        
# Build network
        
network = input_data(shape=[None, 32, 32, 3], dtype=tf.float32)
        
network = conv_2d(network, 32, 3, activation='relu')
        
network = max_pool_2d(network, 2)
        
network = conv_2d(network, 64, 3, activation='relu')
        
network = conv_2d(network, 64, 3, activation='relu')
        
network = max_pool_2d(network, 2)
        
network = fully_connected(network, 512, activation='relu')
        
network = dropout(network, 0.5)
        
network = fully_connected(network, 10, activation='softmax')
        
network = regression(network, optimizer='adam',
        
                     loss='categorical_crossentropy',
        
                     learning_rate=0.001)
        

        
# Training
        
model = tflearn.DNN(network, tensorboard_verbose=0)
        
model.fit(X, Y, n_epoch=50, shuffle=True, validation_set=(X_test, Y_test),
        
          show_metric=True, batch_size=96, run_id='cifar10_cnn')
        

        
h5f.close()
       
       
        1.
        2.
        3.
        4.
        5.
        6.
        7.
        8.
        9.
        10.
        11.
        12.
        13.
        14.
        15.
        16.
        17.
        18.
        19.
        20.
        21.
        22.
        23.
        24.
        25.
        26.
        27.
        28.
        29.
        30.
        31.
        32.
        33.
        34.
        35.
        36.
        37.
        38.
        39.
        40.
        41.
        42.
        43.
        44.
        45.
        46.
        47.
        48.
        49.
        50.
        51.
        52.
        53.
        54.
        55.
        56.
        57.
        58.
        59.
        60.