# Extract object from image and apply it to different image

SHORT BACKGROUND:

I am developing a neural network based model to differentiate benign and malignant ovarian tumors. I am working with transvaginal ultrasound images, some of which contains yellow markers as seen in the first image below. Without going into too mach details, there is a risk that the presence of the markers could introduce a bias that the network will be able to exploit. As a way of trying to counter this bias, I will insert semi-random markers to the images during training and validation as a type of image augmentation.

PROBLEM:

During training, I import the images (stored as JPEG-files) and convert them to NumPy arrays.

• Simply overlaying a marker is not good enough, since the marker would have a clear/sharp outline, while the "real"/already present markers are somewhat distorted due to anti-aliasing/JPEG-image compression.

I have some identical images with/without markers (e.g. first and second image below). After importing the two images as NumPy arrays the marker can be extracted by

• Option 1: ... subtracting one image from the other (see the third and fourth image below for the result). (Would result in incorrect pixel values for the marker.)

• Option 2: ... copying the image with markers and setting all cells to zero where the two images (with/without markers) are (almost) equal (see the fifth and sixth image below for the result).

a)

Let's say, for simplicity, that the only thing that I wanted to achieve was to apply the extracted marker to a different transvaginal ultrasound image. How would I do that?

• Simply overlaying the marker would create black patches around the marker.

• Simply adding the pixel values would result in incorrect pixel values for the marker.

b) (given that a) is solved)

Let's say that I now also want to be able to insert a marker of a different size and orientation.

Orientation: I could rotate and move the marker. Size: Any suggestion of how I could go about doing this? I still want the marker to have the same properties, i.e. simply stretching is not an option.

from keras.preprocessing.image import load_img, img_to_array
from scipy.ndimage import rotate
import numpy as np

# importing images as NumPy arrays

# option 1
img3 = img2-img1   # extracting the marker by subtracting the image arrays
img3 = img3[:-100] # getting rid of the yellow/grey marker in the down-right corner

# option 2
img3 = img2[np.abs(img2 - img1) < 15] = 0

# rotating the image
img3 = rotate(img3, angle=-23.6)

# cropping the image
img3_copy = img3.copy()
img3_copy[np.abs(img3_copy) < 10**-4] = 0
idx = np.nonzero(img3_copy)
img3 = img3[np.min(idx[0]):np.max(idx[0])+1, np.min(idx[1]):np.max(idx[1])+1]