---

The workflow is used in the same way as the example presented in section 2 to assist researchers in investigating the use of dual-modality breast imaging methods combining ultrasound and microwaves. The workflow is applied to an image reconstructed by an enhanced variant of the time-shift-and-sum approach that is applied to numerical ultra-wideband (UWB) microwave radar data. In particular, prior structural information acquired from ultrasound imagery is incorporated into time-delay computations that are used by the reconstruction operator. Most time-shift-and-sum reconstruction algorithms model the propagation path between a focal point and an antenna with a path ray that assumes that the breast interior is homogeneous (see the examples in sections 5 and 6, for example). The approach used in [19] models the path ray with multiple segments: immersion medium, skin, fat, and fibroglandular. The path ray model, the time delay calculations, and the reconstruction operator are described in more detail in section II of [19].

For the example presented in this section, the researcher tests the general hypothesis that the use of prior information to refine the segmentation of the path ray, improves the accuracy of the time delays that are calculated. This, in turn, leads to an improvement in the quality of the reconstructed images. The workflow is used to process a reconstruction and quantify information extracted from regions within the image related to a dominant scatterer corresponding to malignant tissue. The quantitative information about the experiment is used by the researcher to either support or refute the hypothesis.

The researcher sets up a numerical experiment to test the hypothesis. Data are generated from a numerical model of a fatty breast with one tumor embedded in the glandular tissue described in [9]. The relative permittivity component of the electromagnetic model that was used to generate the numerical data serves as ground truth. For the analysis, it is used as the reference image while the backscattered energy image serves as the test image. The reference and test images that are input to the workflow are shown in Figure 1. The metrics are used to quantify differences between the dominant scattering region that is reconstructed, and the tumor region within the forward model.

Figure 1 Relative permittivity component of the electromagnetic model of the fatty breast is used as the reference image (left) while the backscatter energy image reconstructed from the numerical backscattered fields generated from electromagnetic model is the test image (right) [9].

The reference and test images shown in Figure 1 are input to a workflow shown in Figure 2. The workflow is comprised of segmentation and analysis steps. In the first step of the segmentation process, the reference image is partitioned into immersion medium, breast interior, and a region-of-interest. The region-of-interest is partitioned further into malignant and non-malignant tissue regions. The region-of-interest of the reference image is partitioned with a thresholding segmentation technique; an unsupervised machine learning technique is applied to the region-of-interest of the test image. The segmentation steps are described in more detail in section 3.1. The analysis steps are described in more detail in sections 3.2-3.3.

Figure 2a. Segmentation and image analysis workflow.

Figure 2b. Workflow applied to test-reference pair. Images are segmented into dominant scatterers to construct masks. Masks are used to measure discrepancies between regions within test and reference images.

3.1 Segmentation of images

---

3.1.1 Construction of immersion medium, breast interior, and region-of-interest masks

---

The first step of the workflow is to partition the reference image into immersion medium, breast interior, and a region-of-interest. A closed curve delineating the skin surface from the immersion medium is formed. A mask of the breast interior is constructed from the region bound by the curve, and the region outside the breast interior is represented by a mask of the immersion medium.

Next, a region-on-interest (ROI) mask is constructed by uniformly contracting the breast interior mask by some amount (e.g., 3.5 mm) using the morphological contraction method described in [13, 14]. Note that the value of the contraction is adjusted by the user. A value of 3.0 mm is used for this example, but the value that is selected typically exceeds the skin thickness employed in the reference image.

The reference and test images share the same immersion medium, breast interior, and region-of-interest masks. For both the reference and test images, only the region of the breast that is within the region-of-interest (i.e., interior to the skin) is partitioned into tissue types, and is discussed in the following sections.

3.1.2 Segmentation of region-of-interest within reference image

---

The region-of-interest within the reference image is delineated into malignant and non-malignant tissues. The regions containing these tissues are segmented using a simple thresholding segmentation procedure described in [12] (section IV.A). The mask containing malignant tissues for the reference image is formed first by calculating the relative permittivity of the Debye model corresponding to malignant tissue at the incident field frequency as documented in [15]. This value establishes the threshold to segment the malignant tissue within the breast interior. Namely, all pixel values greater than the threshold value within the region-of-interest are classified as malignant tissue. The remaining tissues are classified as non-malignant tissues to construct the non-malignant tissues mask. The malignant and non-malignant tissue masks formed from the region-of-interest using this technique are shown in the left column of Figure 3. The breast interior mask formed in the first step is also shown in the top row of Figure 3.

Each mask is mapped to a tissue type. When using the threshold technique, each tissue type is assumed known based on the threshold value applied. Therefore, mapping the segmented masks to the corresponding tissue type is straightforward. Namely, the mask corresponding to malignant tissue is mapped to the tumor label, and the non-malignant tissue mask is mapped to the non-malignant label. Note that a skin mask is formed by subtracting the region-of-interest mask from the breast interior mask. The skin mask is mapped to the skin tissue label. Finally, the immersion mask is mapped to the background. The disjoint union of the tissue type regions leads to a tissue type image shown in the left panel of the middle row of Figure 4.

Figure 3. Masks constructed by segmentation techniques applied to the region-of-interest within the image. The reference masks (left column) are constructed with a thresholding technique and the test masks (right column) are constructed with an unsupervised machine learning technique.

Figure 4. Reference and test images for a fatty breast (first row) with the corresponding tissue types identified via segmentation (middle row). Cluster map for final iteration of segmentation of region-of-interest of test image (bottom row).

3.1.3 Segmentation of region-of-interest within test image

---

An automatic segmentation technique is applied to the region-of-interest within the test image to partition the region based on the intensity of the reconstructed backscatter energy. It is comprised of an iterative unsupervised machine learning approach and is described in [17, 18]. First, the region-of-interest mask is applied to the test image and the automatic segmentation algorithm applies the k-means clustering algorithm to the extracted region in an iterative fashion. After each iteration, a statistical test evaluates the hypothesis that the cumulative distribution function of the pixel/voxel values within clusters at the previous and current iterations originate from different distributions. If the hypothesis does not fail, then the number of clusters used by the algorithm is incremented, otherwise convergence is implied and the image is not partitioned further. The iterative procedure is summarized by the flow diagram presented in Figure 5 of 2.1.3.

The segmentation results of the region-of-interest within the test image after each iteration of algorithm are shown in Figure 5. For this example, when the segmentation algorithm is terminated, the region-of-interest within the test image is partitioned into $6$ clusters. The cluster map for the final iteration of the segmentation algorithm is shown in the right hand side of the bottom row of Figure 4.

Next, a mapping function assigns the clusters to binary images that have the same dimensions as the test image and are referred to as masks. Given the region-of-interest has been partitioned into $k$ clusters, the highest valued clusters, $c_{max(k)}$ correlates to the dominant scatterer (i.e., highest backscatter energy), so they are mapped to the Malignant tissue mask. Likewise, cluster $c_{1}$ identifies the Background and correlates to pixels outside the region-of-interest. The remaining clusters $c_{2}$ to $c_{max(k)-1}$ are mapped to the Non-malignant tissue mask.

Similar to what was described for the reference image, a skin mask is formed from the region bound by the skin surface contour and the contour that bounds the region-of-interest mask. The background and skin masks are not used by the analysis component of the workflow. They are only used to form the tissue type image, which is used as a visualization tool. The cluster-to-mask mapping function that maps clusters, $c_{k}$ where $k = 1, 2, …, max(k)$, to the corresponding mask is defined as,

Each tissue mask is mapped to a tissue type. The mask corresponding to malignant tissue is mapped to the tumor label, and the non-malignant tissue mask is mapped to the non-malignant label. The skin mask is mapped to the skin tissue type and the background mask is mapped to the background. The disjoint union of the tissue type regions leads to a tissue type image shown in middle of the right column of Figure 4. The tissue type images constructed from the reference and test images are used for qualitative purposes and can assist with the interpretation of the backscatter energy images. Conversely, performance metrics are applied to the tissue masks and are used for the quantitative analysis of the backscatter energy images. Note that the immersion medium and skin masks are only used to construct the tissue type image; metrics are not applied to these masks in the quantitative analysis steps.

Figure 5 Segmentation results of the test image after each iteration of algorithm. Region-of-interest within test image is partitioned into 6 clusters when convergence is sensed.

3.2 Interface analysis of fatty breast images

---

The edge points that are sampled uniformly from the boundary of the test masks constructed by the segmentation task collectively represent estimated points on interfaces of regions of significant scattering. A contour is fitted to the estimated points and superimposed onto the ground truth model along with the corresponding contour of the reference mask. The contours are also superimposed onto the backscatter energy map. The ground truth model and the backscatter energy map adds context to the test and reference contours, so the images are helpful to visually evaluate the quality of the interface extracted from the data. These images are shown in the top row of Figure 6. The reference contour of the tumor region is shown in red while the test contour of the significant scattering region is represented with the blue line.

The estimated points sampled from the boundary of a test mask are compared with the corresponding boundary of the reference mask to measure how accurately the interface of the significant scattering region is extracted by the reconstruction algorithm. More specifically, the ability of the reconstruction algorithm to accurately extract a tumor interface is evaluated by measuring the distance each edge point on a reconstructed interface is to the nearest point on the corresponding reference surface (refer to section 1 description of metrics). A negative distance implies that the estimated point is within the ground truth tumor region, while a positive distance implies the opposite.

In the bottom center panels of Figure 6, points of the estimated interface along with the errors are superimposed onto the reference contour to help visualize how the errors are spatially distributed along the interface. A quantitative analysis of the errors is provided by plotting the error distribution along with an estimate of the mean and variance (or spread) of the errors. This information is shown in the bottom right hand panel of Figure 6.

Figure 6. Interface analysis results of interface associated with dominant scattering region.

3.2 Geometric property and backscatter energy analysis of fatty breast images

---

The significant scattering region is examined to determine if it is connected and there is overlap with the reference tumor region. If it is, then a comprehensive geometric property analysis is performed to assess the overlap between the reconstructed significant scattering region using metrics 4-8 and 16 described in the Description of Metrics. The overlap metrics 4-8 are primarily used to measure how accurately the scattering region is reconstructed relative to the reference region, so the measured values are most meaningful when the ground truth is used as the reference image (as in this case).

A suite of metrics is added to the workflow to support backscatter energy analysis, and is described in more detail in section 1.4.a. To analyze backscatter energy, the test masks are applied to the image to extract the energy bounded by the contour of the mask. The resulting energy is assessed with metrics 20-22 (localization error, FWHM, and SMR). The results of the analysis when the metrics are applied to the reconstructed significant scattering region are shown in the table within Figure 7.

Figure 7. Dominant scattering region of backscatter energy image compared with tumor tissue region within reference image.