The training speed is currently bottlenecked by the data input. The training on the GPU is faster than the data preprocessing on the CPU. The Batch Input Queue is therefore always empty. Improving the data input can increase the training speed by up to a factor of two.
I have tracked down the performance error to the data augmentation.
One solution might be to do the data augmentation on whole image instead of individual patches.
The training speed is currently bottlenecked by the data input. The training on the GPU is faster than the data preprocessing on the CPU. The Batch Input Queue is therefore always empty. Improving the data input can increase the training speed by up to a factor of two.
I have tracked down the performance error to the data augmentation.
One solution might be to do the data augmentation on whole image instead of individual patches.