[keras] train py

Author code ：zishang33/3DUnetCNN
This paper mainly analyzes train.py The role of the various parts of ．

main function

From the process of operation , When running, put config[“overwrite”] Change it to true.
Model calls are not analyzed here load_old_model The situation of .

The first code data fetch

if overwrite or not os.path.exists(config["data_file"]):
    training_files = fetch_training_data_files()
    write_data_to_file(training_files, config["data_file"], image_shape=config["image_shape"])
data_file_opened = open_data_file(config["data_file"])

Line by line

training_files = fetch_training_data_files()

config[“overwrite”] = True stay main Call the calling function in fetch_training_data_files
Put all the nii File paths are saved in training_files in
Look at the following fetch_training_data_files() Detailed explanation

training_files Tuples containing training data files tuple list . In each tuple tuple in , Several modes should be listed in the same order . The last item in each tuple must be a tagged image （truth）.
for example ：
[(‘sub1-T1.nii.gz’, ‘sub1-T2.nii.gz’, ‘sub1-truth.nii.gz’),
(‘sub2-T1.nii.gz’, ‘sub2-T2.nii.gz’, ‘sub2-truth.nii.gz’)]

write_data_to_file(training_files, config["data_file"], image_shape=config["image_shape"])

write_data_to_file The function is to receive a set of training images and write these images into hdf5 file , stay data Specific analysis in the document

config[“data_file”] It's about putting hdf5 Where the file is written . The definition for

config["data_file"] = os.path.abspath("brats_data.h5")# Return to absolute path 

data_file_opened = open_data_file(config["data_file"])

Finally, use the function open_data_file() Read table File data ．

The second code Generate data generators

train_generator, validation_generator, n_train_steps, n_validation_steps = get_training_and_validation_generators(
    data_file_opened,
    batch_size=config["batch_size"],
    data_split=config["validation_split"],
    overwrite=overwrite,
    validation_keys_file=config["validation_file"],
    training_keys_file=config["training_file"],
    n_labels=config["n_labels"],
    labels=config["labels"],
    patch_shape=config["patch_shape"],
    validation_batch_size=config["validation_batch_size"],
    validation_patch_overlap=config["validation_patch_overlap"],
    training_patch_start_offset=config["training_patch_start_offset"],
    permute=config["permute"],
    augment=config["augment"],
    skip_blank=config["skip_blank"],
    augment_flip=config["flip"],
    augment_distortion_factor=config["distort"])

Utilization function get_training_and_validation_generators() Pack training data and test data into Keras Input data of frame type . For future network training fit_generator Function ready , Don't understand fit_generator You can read my blog before . This function will be analyzed in its definition file .

The third code Training network

# run training
    train_model(model=model,
                model_file=config["model_file"],
                training_generator=train_generator,
                validation_generator=validation_generator,
                steps_per_epoch=n_train_steps,
                validation_steps=n_validation_steps,
                initial_learning_rate=config["initial_learning_rate"],
                learning_rate_drop=config["learning_rate_drop"],
                learning_rate_patience=config["patience"],
                early_stopping_patience=config["early_stop"],
                n_epochs=config["n_epochs"])

This piece is nothing ,train_model A function is actually a call to fit_generator Function to complete our network training , among model For the definition of

model = unet_model_3d(input_shape=config["input_shape"],
                          pool_size=config["pool_size"],
                          n_labels=config["n_labels"],
                          initial_learning_rate=config["initial_learning_rate"],
                          deconvolution=config["deconvolution"])

fetch_training_data_files

Back to all preprocessed In all nii Path to file

---

Some file handling functions

1. glob File name pattern matching , Do not traverse the entire directory to determine whether each file is consistent with .

  import glob
  # Query files with subdirectories 
  print ('Named explicitly:')
  for name in glob.glob('dir/subdir/*'):
      print ('\t', name)
  # Use wildcards *  Replace subdirectory name 
  print ('Named with wildcard:')
  for name in glob.glob('dir/*/*'):
      print ('\t', name)

  # Output 

  Named explicitly:
          dir/subdir/subfile.txt
  Named with wildcard:
          dir/subdir/subfile.txt

2. os.path.join() function ： Connect two or more pathname components

    import os

    Path1 = 'home'
    Path2 = 'develop'
    Path3 = 'code'

    Path10 = Path1 + Path2 + Path3
    Path20 = os.path.join(Path1,Path2,Path3)
    print ('Path10 = ',Path10)
    print ('Path20 = ',Path20)

    # Output 

    Path10 = homedevelopcode
    Path20 = home\develop\code

3. os.path.dirname(path) function ： Remove the filename , Return directory

   __file__ Represents the of the current file path
   os.path.dirname((__file__) Is to get the absolute path of the current file 

Function specific analysis

training_data_files = list()

Create a list To save all to be processed nii Path to file

for subject_dir in glob.glob(os.path.join(os.path.dirname(__file__), "data", "preprocessed", "*", "*")):

First use os.path.dirname Returns the current file （ clip ） The absolute path of , Reuse join Put him and data,preprocessed Connect , Last use glob Find all the image folders under it , And loop through

subject_files = list()

Create a sub file path to save preprocessed In each image folder under the folder nii Path to file

config["all_modalities"] = ["t1", "t1ce", "flair", "t2"]
config["training_modalities"] = config["all_modalities"]

Four kinds of data of training set

for modality in config["training_modalities"] + ["truth"]:

amount to preprocessed In the image folder of nii The five forms of ：“t1”, “t1ce”, “flair”, “t2”,“truth”

subject_files.append(os.path.join(subject_dir, modality + ".nii.gz"))

Record the five nii Path to file

training_data_files.append(tuple(subject_files))

Add all paths to training_data_files Inside and back

Function output analysis

Try to run fetch_training_data_files And observe its output , I've only intercepted part of it here preprocessed File to observe .

training_files = fetch_training_data_files()
print(training_files)

Output

[('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0033/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0033/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0033/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0033/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0033/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0027/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0027/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0027/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0027/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0027/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0009/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0009/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0009/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0009/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0009/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0011/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0011/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0011/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0011/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0011/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0069/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0069/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0069/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0069/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0069/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0034/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0034/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0034/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0034/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0034/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0064/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0064/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0064/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0064/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0064/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0047/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0047/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0047/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0047/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0047/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0046/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0046/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0046/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0046/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0046/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0037/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0037/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0037/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0037/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0037/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0059/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0059/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0059/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0059/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0059/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0068/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0068/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0068/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0068/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0068/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0070/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0070/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0070/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0070/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0070/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0075/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0075/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0075/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0075/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0075/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5393/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5393/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5393/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5393/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5393/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5397/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5397/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5397/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5397/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5397/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4942/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4942/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4942/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4942/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4942/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6188/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6188/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6188/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6188/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6188/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5396/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5396/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5396/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5396/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-5396/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6666/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6666/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6666/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6666/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6666/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6668/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6668/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6668/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6668/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6668/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6665/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6665/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6665/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6665/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6665/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4944/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4944/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4944/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4944/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-4944/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6186/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6186/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6186/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6186/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6186/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5851/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5851/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5851/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5851/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5851/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5855/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5855/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5855/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5855/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5855/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6669/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6669/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6669/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6669/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-CS-6669/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5854/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5854/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5854/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5854/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5854/truth.nii.gz'), ('data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5872/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5872/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5872/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5872/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_LGG_NIfTI_and_Segmentations/TCGA-DU-5872/truth.nii.gz')]

Let's select one of the tuples to see

('data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t1.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t1ce.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/flair.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/t2.nii.gz', 'data/preprocessed/Pre-operative_TCGA_GBM_NIfTI_and_Segmentations/TCGA-02-0006/truth.nii.gz')

The visible paths are all by t1,tice,flair,t2,truth In order
And its total length （list Number of middle element groups ） by preprocessed The total number of image folders to be processed

print(type(training_files))
print(len(training_files))

<class 'list'>
30