Transfer learning to freeze the network:

Description: pytorch (1-3), TensorFlow (4)

fine tune is to freeze the front layers of the network, and then train the last layer

1. Pass all parameters to the optimizer, but set the requires_grad of the parameters of the layer to be frozen to False:

   optimizer = optim.SGD(model.parameters(), lr=1e-2) # All parameters are passed infor name, param in model.named_parameters():if the name of the network layer to be frozen (that is, the value of name):param.requires_grad = False

2. The optimizer passes in the parameters of the unfrozen network layer:

   optimizer = optim.SGD(model.name.parameters() of the unfrozen network layer, lr=1e-2) # The optimizer only passes in the parameters of fc2 

3. The best practice is: the optimizer only passes in the parameter of requires_grad=True, the memory occupied will be smaller, and the efficiency will be higher.Code and Combining 1 and 2

   1. Save video memory: do not pass parameters that will not be updated to optimizer

   2. Increase the speed: set the requires_grad of the parameters that are not updated to False, saving the time to calculate the gradient of these parameters

4. The code is as follows:

   #Define optimization operatoroptimizer = tf.train.AdamOptimizer(1e-3)#Select the parameters to be optimizedoutput_vars = tf.get_collection(tf.GraphKyes.TRAINABLE_VARIABLES, scope= 'outpt')train_step = optimizer.minimize(loss_score,var_list = output_vars)

   
   Put the layer that needs to update the gradient in the tf.get_collection function, and do not put it in if it does not need to be updated.
   1. The main function of the function: from a collectionretrieve variable
   2. is used to get all the elements in the key collection and return a list.The order of the list is based on the order in which the variables were placed in the set.scope is an optional parameter, indicating the namespace (namedomain), if specified, it will return a list of all variables put into 'key' in the name domain (for example, the outpt description in the sample code is the parameter to return the outpt layer), if not specified, it will return all variables.