Model Training

model_training¶

Description

This module contains functions to train a VGG16 encoder U-net decoder CNN. The module was specifically designed to be executed from a GUI. When used from the GUI, the module saves the trained model and weights to a given directory. The user needs to provide paths to the image and label/ mask directories. Instructions for correct image labelling can be found in the Labelling directory.

Functions scope

conv_block Function to build a convolutional block for the U-net decoder path of the network. The block is built using several keras.layers functionalities. decoder_block Function to build a decoder block for the U-net decoder path of the network. The block is built using several keras.layers functionalities. build_vgg16_model Function that builds a convolutional network consisting of an VGG16 encoder path and a U-net decoder path. IoU Function to compute the intersection over union score (IoU), a measure of prediction accuracy. This is sometimes also called Jaccard score. dice_score Function to compute the Dice score, a measure of prediction accuracy. focal_loss Function to compute the focal loss, a measure of prediction accuracy. load_images Function to load images and manually labeled masks from a specified directory. train_model Function to train a convolutional neural network with VGG16 encoder and U-net decoder. All the steps necessary to properly train a neural network are included in this function.

Notes

Additional information and usage examples can be found at the respective functions documentations.

`StopTrainingCallback` ¶

Bases: Callback

A custom Keras callback to stop training early based on a GUI signal.

This callback checks the should_stop attribute of a GUI instance at the beginning of each epoch and stops training if the flag is set to True.

PARAMETER	DESCRIPTION
`gui_instance`	The GUI instance that contains the `should_stop` attribute. It should be set to `True` when the user requests training to stop. TYPE: `object`

ATTRIBUTE	DESCRIPTION
`gui_instance`	Reference to the GUI instance for checking the stopping condition. TYPE: `object`

METHOD	DESCRIPTION
`on_epoch_begin`	Checks if `should_stop` is set to `True` and stops training if necessary.

Examples:

>>> from tensorflow.keras.models import Sequential
>>> from tensorflow.keras.layers import Dense
>>> model = Sequential([Dense(10, activation='relu', input_shape=(20,))])
>>> model.compile(optimizer='adam', loss='mse')
>>> gui_instance.should_stop = False  # Simulating a GUI stop flag
>>> stop_callback = StopTrainingCallback(gui_instance)
>>> model.fit(X_train, y_train, epochs=100, callbacks=[stop_callback])

`init(gui_instance)` ¶

PARAMETER	DESCRIPTION
`gui_instance`	The GUI instance containing the `should_stop` attribute. TYPE: `object`

`on_epoch_begin(epoch, logs=None)` ¶

Called at the beginning of each epoch. Checks if training should stop.

PARAMETER	DESCRIPTION
`epoch`	The index of the current epoch. TYPE: `int`
`logs`	Dictionary containing training metrics (default is None). TYPE: `dict` DEFAULT: `None`

`IoU(y_true, y_pred, dtype=tf.float32)` ¶

Function to compute the intersection over union score (IoU), a measure of prediction accuracy. This is sometimes also called Jaccard score.

The IoU can be used as a loss metric during binary segmentation when convolutional neural networks are applied. The IoU is calculated for both the training and validation set.

PARAMETER	DESCRIPTION
`y_true`	True positive image segmentation label predefined by the user. This is the mask that is provided prior to model training. TYPE: `Tensor`
`y_pred`	Predicted image segmentation by the network. TYPE: `Tensor`
`dtype`	Data type of the IoU calculation. The default is tf.float32. TYPE: `default = tff.float32` DEFAULT: `float32`

RETURNS	DESCRIPTION
`iou`	IoU representation in the same shape as y_true, y_pred. TYPE: `Tensor`

Notes

The IoU is usually calculated as IoU = intersection / union. The intersection is calculated as the overlap of y_true and y_pred, whereas the union is the sum of y_true and y_pred.

Examples:

>>> IoU(y_true=Tensor("IteratorGetNext:1", shape=(1, 512, 512, 1), dtype=float32),
        y_pred=Tensor("VGG16_U-Net/conv2d_8/Sigmoid:0",
        shape=(1, 512, 512, 1), dtype=float32), dtype=float32)
Tensor("truediv:0", shape=(1, 512, 512), dtype=float32)

`build_vgg16_unet(input_shape)` ¶

Function that builds a convolutional network consisting of a VGG16 encoder path and a U-net decoder path.

The model is built using several Tensorflow.Keras functions. First, the whole VGG16 model is imported and built using pretrained imagenet weights and the input shape. Then, the encoder layers are pulled from the model incldung the bridge. Subsequently the decoder path from the U-net is built. Lastly, a 1x1 convolution is applied with sigmoid activation to perform binary segmentation on the input.

PARAMETER	DESCRIPTION
`input_shape`	Tuple describing the input shape. Must be of shape (...,...,...). Here we used (512,512,3) as input shape. The image size (512,512,) can be easily adapted. The channel numer (,,3) is given by the model and the pretrained weights. We advide the user not to change the image size segmentation results were best with the predefined size. TYPE: `tuple`

RETURNS	DESCRIPTION
`model`	The built VGG16 encoder U-net decoder convolutional network for binary segmentation on the input. The model can subsequently be used for training.

Notes

See our paper () and references for more detailed model description

References

[1] VGG16: Simonyan, Karen, and Andrew Zisserman. “Very deep convolutional networks for large-scale image recognition.” arXiv preprint arXiv:1409.1556 (2014) [2] U-net: Ronneberger, O., Fischer, P. and Brox, T. "U-Net: Convolutional Networks for Biomedical Image Segmentation." arXiv preprint arXiv:1505.04597 (2015)

`conv_block(inputs, num_filters)` ¶

Function to build a convolutional block for the U-net decoder path of the network to be build. The block is built using several keras.layers functionalities.

Here, we decided to use 'padding = same' and and a convolutional kernel of 3. This is adaptable in the code but will influence the model outcome. The convolutional block consists of two convolutional layers. Each creates a convolution kernel that is convolved with the layer input to produce a tensor of outputs.

PARAMETER	DESCRIPTION
`inputs`	Concattenated Tensorflow.Keras Tensor outputted from previous layer. The Tensor can be altered by adapting, i.e. the filter numbers but this will change the model training output. The input is then convolved using the built kernel. TYPE: `KerasTensor`
`num_filters`	Integer variable determining the number of filters used during model training. Here, we started with 'num_filers = 512'. The filter number is halfed each layer. The number of filters can be adapted in the code. Must be non-negative and non-zero. TYPE: `int`

RETURNS	DESCRIPTION
`x`	Tensorflow.Keras Tensor used during model Training. The Tensor can be altered by adapting the input paramenters to the function or the upsampling but this will change the model training. The number of filters is halfed. TYPE: `KerasTensor`

Examples:

>>> conv_block(inputs=KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 128),
               dtype=tf.float32, name=None),
               num_filters=128)
KerasTensor(type_spec=TensorSpec(shape=(None, 256, 256, 64),
dtype=tf.float32, name=None)

`decoder_block(inputs, skip_features, num_filters)` ¶

Function to build a decoder block for the U-net decoder path of the network to be build. The block is build using several keras.layers functionalities.

The block is built by applying a deconvolution (Keras.Conv2DTranspose) to upsample to input by a factor of 2. A concatenation with the skipped features from the mirrored vgg16 convolutional layer follows. Subsequently a convolutional block (see conv_block function) is applied to convolve the input with the built kernel.

PARAMETER	DESCRIPTION
`inputs`	Concattenated Tensorflow.Keras Tensor outputted from previous layer. The Tensor can be altered by adapting, i.e. the filter numbers but this will change the model training output. TYPE: `KerasTensor`
`skip_features`	Skip connections to the encoder path of the vgg16 encoder. TYPE: `Keras Tensor`
`num_filters`	Integer variable determining the number of filters used during model training. Here, we started with 'num_filers = 512'. The filter number is halfed each layer. The number of filters can be adapted in the code. Must be non-neagtive and non-zero. TYPE: `int`

RETURNS DESCRIPTION

x

Tensorflow.Keras Tensor used during model Training. The tensor is upsampled using Keras.Conv2DTranspose with a kernel of (2,2), 'stride=2' and 'padding=same'. The upsampling increases image size by a factor of 2. The number of filters is halfed. The Tensor can be altered by adapting the input paramenters to the function or the upsampling but this will change the model training.

TYPE: KerasTensor

Examples:

>>> decoder_block(inputs=KerasTensor(type_spec=TensorSpec(shape=(None, 64, 64, 512),
                  skip_features=KerasTensor(type_spec=TensorSpec(shape=(None, 64, 64, 512),
                  dtype=tf.float32, name=None)),
                  num_filters=256)
KerasTensor(type_spec=TensorSpec(shape=(None, 128, 128, 256),
dtype=tf.float32, name=None)

`dice_score(y_true, y_pred)` ¶

Function to compute the Dice score, a measure of prediction accuracy.

The Dice score can be used as a loss metric during binary segmentation when convolutional neural networks are applied. The Dice score is calculated for both the training and validation set.

PARAMETER	DESCRIPTION
`y_true`	True positive image segmentation label predefined by the user. This is the mask that is provided prior to model training. TYPE: `Tensor`
`y_pred`	Predicted image segmentation by the network. TYPE: `Tensor`

RETURNS	DESCRIPTION
`score`	Dice score representation in the same shape as y_true, y_pred. TYPE: `Tensor`

Notes

The IoU is usually calculated as Dice = 2 * intersection / union. The intersection is calculated as the overlap of y_true and y_pred, whereas the union is the sum of y_true and y_pred.

Examples:

>>> IoU(y_true=Tensor("IteratorGetNext:1", shape=(1, 512, 512, 1),
        dtype=float32),
        y_pred=Tensor("VGG16_U-Net/conv2d_8/Sigmoid:0",
        shape=(1, 512, 512, 1), dtype=float32),
        smooth=1)
Tensor("dice_score/truediv:0", shape=(1, 512, 512), dtype=float32)

`focal_loss(y_true, y_pred, alpha=0.8, gamma=2)` ¶

Function to compute the focal loss, a measure of prediction accuracy.

The focal loss can be used as a loss metric during binary segmentation when convolutional neural networks are applied. The focal loss score is calculated for both, the training and validation set. The focal loss is specifically applicable when class imbalances, i.e. between foregroung (muscle aponeurosis) and background (not muscle aponeurosis), are existent.

PARAMETER	DESCRIPTION
`y_true`	True positive image segmentation label predefined by the user. This is the mask that is provided prior to model training. TYPE: `Tensor`
`y_pred`	Predicted image segmentation by the network. TYPE: `Tensor`
`alpha`	Coefficient used on positive exaples, must be non-negative and non-zero. TYPE: `float` DEFAULT: `= 0.8`
`gamma`	Focussing parameter, must be non-negative and non-zero. TYPE: `float` DEFAULT: `= 2`

RETURNS	DESCRIPTION
`f_loss`	Tensor containing the calculated focal loss score. TYPE: `Tensor`

Examples:

>>> focal_loss(y_true=Tensor("IteratorGetNext:1", shape=(1, 512, 512, 1),
        dtype=float32),
        y_pred=Tensor("VGG16_U-Net/conv2d_8/Sigmoid:0",
        shape=(1, 512, 512, 1), dtype=float32),
        smooth=1)
Tensor("focal_loss/Mean:0", shape=(), dtype=float32)

`loadImages(img_path, mask_path)` ¶

Function to load images and manually labeled masks from a specified directory.

The images and masks are loaded, resized and normalized in order to be suitable and usable for model training. The specified directories must lead to the images and masks. The number of images and masks must be equal. The images and masks can be in any common image format. The names of the images and masks must match. The image and corresponding mask must have the same name.

PARAMETER	DESCRIPTION
`img_path`	Path that leads to the directory containing the training images. Image must be in RGB format. TYPE: `str`
`mask_path`	Path that leads to the directory containing the mask images. Masks must be binary. TYPE: `str`

RETURNS	DESCRIPTION
`train_imgs`	Resized, normalized training images stored in a numpy array. TYPE: `ndarray`
`mask_imgs`	Resized, normalized training masks stored in a numpy array. TYPE: `ndarray`

Notes

See labelling instruction for correct masks creation and use, if needed, the supplied ImageJ script to label your images.

Examples:

>>> loadImages(img_path = "C:/Users/admin/Dokuments/images",
               mask_path = "C:/Users/admin/Dokuments/masks")
train_imgs([[[[0.22414216 0.19730392 0.22414216] ... [0.22414216 0.19730392 0.22414216]]])
mask_imgs([[[[0.] ... [0.]]])

`trainModel(img_path, mask_path, out_path, batch_size, learning_rate, epochs, loss, gui)` ¶

Function to train a convolutional neural network with VGG16 encoder and U-net decoder. All the steps necessary to properly train an neural network are included in this function.

This functions build upon all the other functions included in this module. Given that all input parameters are correctly specified, the images and masks are loaded, splittet into test and training sets, the model is compiled according to user specification and the model is trained.

PARAMETER	DESCRIPTION
`img_path`	Path that leads to the directory containing the training images. Image must be in RGB format. TYPE: `str`
`mask_path`	Path that leads to the directory containing the mask images. Masks must be binary. TYPE: `str`
`out_path`	Path that leads to the directory where the trained model is saved. TYPE: `str`
`batch_size`	Integer value that determines the batch size per iteration through the network during model training. Although a larger batch size has advantages during model trainig, the images used here are large. Thus, the larger the batch size, the more compute power is needed or the longer the training duration. Must be non-negative and non-zero. TYPE: `int`
`learning_rate`	Float value determining the learning rate used during model training. Must be non-negative and non-zero. TYPE: `float`
`epochs`	Integer value that determines the amount of epochs that the model is trained befor training is aborted. The total amount of epochs will only be used if early stopping does not happen. Must be non-negative and non-zero. TYPE: `int`
`loss`	String variable that determines the loss function used during training. So far, only one type is supported here: - Binary cross-entropy. loss == "BCE" TYPE: `BCE` DEFAULT: `"BCE"`
`gui`	A tkinter.TK class instance that represents a GUI. By passing this argument, interaction with the GUI is possible i.e., stopping the model training model process. TYPE: `TK`

Notes

For specific explanations for the included functions see the respective function docstrings in this module. This function can either be run from the command prompt or is called by the GUI. Note that the functioned was specifically designed to be called from the GUI. Thus, tk.messagebox will pop up when errors are raised even if the GUI is not started.

Examples:

>>> trainModel(img_path= "C:/Users/admin/Dokuments/images",
               mask_path="C:/Users/admin/Dokuments/masks",
               out_path="C:/Users/admin/Dokuments/results",
               batch_size=1, learning_rate=0.005,
               epochs=3, loss="BCE", gui)

Description

Python module which contains a function, which allows to generate new training images from the input images. The newly generated data will be saved under the same directories as the input data.

This module provides a function, image_augmentation, that performs data augmentation on input images and their corresponding masks to generate new training data for machine learning models. Data augmentation is a common technique used to artificially increase the diversity of the training dataset by applying various transformations to the original images.

Functions scope

image_augmentation Function, which generates new training data from the input images through data augmentation.

`image_augmentation(input_img_folder, input_mask_folder, gui)` ¶

Function, which generates new training data from the input images through data augmentation. At the moment the number of added images is set to five. Perform data augmentation on images and masks in the specified input directories.

The function applies data augmentation techniques to the images and masks located in the specified input directories. It creates augmented images and masks based on various augmentation parameters, and saves the augmented images and masks back to their respective input directories.

PARAMETER	DESCRIPTION
`input_img_folder`	Path to the folder containing the original input images. TYPE: `str`
`input_mask_folder`	Path to the folder containing the original input masks corresponding to the images. TYPE: `str`
`gui`	The main tkinter GUI object to display information to the user. TYPE: `Tk`

RETURNS	DESCRIPTION
`None`

Notes

Although nothing is returned, the images in the input folders will be augmented three-fold.
The function uses the Keras ImageDataGenerator for data augmentation.
Augmented images and masks will be saved to their respective input directories with filenames prefixed with numbers representing the index of the original images.
The function will display information to the user in the specified tkinter GUI.

Examples:

>>> root = tk.Tk()
>>> image_augmentation("data/images/", "data/masks/", root

Module to perform analysis of image files used for training model

`collect_images(root_dir, target_dir, image_type)` ¶

Searches through the root directory and its subdirectories for images of the specified type and copies them to the target directory with modified names to avoid overwriting. The modified name format is original_filename_n.extension, where n starts from 0.

PARAMETER	DESCRIPTION
`root_dir`	The root directory to search for images. TYPE: `str`
`target_dir`	The directory where found images will be stored. TYPE: `str`
`image_type`	The type of the images to search for (e.g., 'jpg', 'png', 'tiff'). TYPE: `str`

Notes

The function creates the target directory if it does not already exist.

`find_outliers(dir1, dir2)` ¶

Find image filenames that do not occur in both directories and check if both directories have the same number of images.

PARAMETER	DESCRIPTION
`dir1`	Path to the first directory containing images. TYPE: `str`
`dir2`	Path to the second directory containing images. TYPE: `str`

RETURNS	DESCRIPTION
`DataFrame`	A DataFrame with columns: - 'Outlier Image': Names of the outlier images. - 'Directory': The directory in which the outlier was found. - 'Images in Dir1': The number of images in dir1. - 'Images in Dir2': The number of images in dir2. TYPE: `dataframe`

Examples:

>>> find_outliers('/path/to/dir1', '/path/to/dir2')
              Outlier Image         Directory  Images in Dir1  Images in Dir2
0           image3.jpg      /path/to/dir1            NaN                 NaN
1           image4.jpg      /path/to/dir2            NaN                 NaN
2           NaN                 NaN                     5                   4

`overlay_directory_images(image_dir, mask_dir, alpha=0.5, start_index=0)` ¶

Overlay binary masks on ultrasound images from given directories.

PARAMETER	DESCRIPTION
`image_dir`	Directory containing the ultrasound images. TYPE: `str`
`mask_dir`	Directory containing the corresponding binary masks. TYPE: `str`
`alpha`	Opacity level of the mask when overlaid on the ultrasound. Default is 0.5. TYPE: `float` DEFAULT: `0.5`
`start_index`	Index to start displaying the image/mask pairs. Default is 0 (first image). TYPE: `int` DEFAULT: `0`

RETURNS	DESCRIPTION
`None`	Displays an interactive plot of overlaid image pairs.

Examples:

>>> overlay_directory_images('/path/to/ultrasound_images/', '/path/to/masks/', start_index=2)

`redact_images_in_directory(directory)` ¶

Redacts the upper 50 pixels of every image in the specified directory by drawing a black rectangle over them. The images are saved under the same names in the same directory.

PARAMETER	DESCRIPTION
`directory`	The path to the directory containing the images to be redacted. TYPE: `str`

`show_outliers_popup(df, dir1, dir2)` ¶

Display a pop-up window with a table showing outlier images between two directories.

PARAMETER	DESCRIPTION
`df`	DataFrame containing outlier information. TYPE: `DataFrame`
`dir1`	Path to the first directory containing images. TYPE: `str`
`dir2`	Path to the second directory containing images. TYPE: `str`

RETURNS	DESCRIPTION
`None`

Model Training

model_training¶

StopTrainingCallback ¶

__init__(gui_instance) ¶

on_epoch_begin(epoch, logs=None) ¶

IoU(y_true, y_pred, dtype=tf.float32) ¶

build_vgg16_unet(input_shape) ¶

conv_block(inputs, num_filters) ¶

decoder_block(inputs, skip_features, num_filters) ¶

dice_score(y_true, y_pred) ¶

focal_loss(y_true, y_pred, alpha=0.8, gamma=2) ¶

loadImages(img_path, mask_path) ¶

trainModel(img_path, mask_path, out_path, batch_size, learning_rate, epochs, loss, gui) ¶

image_augmentation(input_img_folder, input_mask_folder, gui) ¶

collect_images(root_dir, target_dir, image_type) ¶

find_outliers(dir1, dir2) ¶

overlay_directory_images(image_dir, mask_dir, alpha=0.5, start_index=0) ¶

redact_images_in_directory(directory) ¶

show_outliers_popup(df, dir1, dir2) ¶

`StopTrainingCallback` ¶

`init(gui_instance)` ¶

`on_epoch_begin(epoch, logs=None)` ¶

`IoU(y_true, y_pred, dtype=tf.float32)` ¶

`build_vgg16_unet(input_shape)` ¶

`conv_block(inputs, num_filters)` ¶

`decoder_block(inputs, skip_features, num_filters)` ¶

`dice_score(y_true, y_pred)` ¶

`focal_loss(y_true, y_pred, alpha=0.8, gamma=2)` ¶

`loadImages(img_path, mask_path)` ¶

`trainModel(img_path, mask_path, out_path, batch_size, learning_rate, epochs, loss, gui)` ¶

`image_augmentation(input_img_folder, input_mask_folder, gui)` ¶

`collect_images(root_dir, target_dir, image_type)` ¶

`find_outliers(dir1, dir2)` ¶

`overlay_directory_images(image_dir, mask_dir, alpha=0.5, start_index=0)` ¶

`redact_images_in_directory(directory)` ¶

`show_outliers_popup(df, dir1, dir2)` ¶