2. Application to microwave tomography : Heterogeneously dense breast example - djkurran/Automated-framework-for-evaluating-microwave-and-multi-modality-breast-images GitHub Wiki
In the first example, the workflow facilitates researchers' investigation of dual-modality breast imaging methods. Internal structural information of the breast interior is acquired from ultrasound imagery. The information is then incorporated as an inhomogeneous numerical background in the quantitative microwave reconstruction algorithm [12]. The researchers conduct a numerical experiment to test the hypothesis that the use of the prior information leads to an improvement in the quality of the reconstruction. Accordingly, the aim of this first example is to demonstrate how the workflow processes images to extract quantitative information about an experiment. The quantitative information is retrieved from specific regions within the images that are dominated by a tissue type.
The images are formed from 2D numerical data of an heterogeneously dense breast having two tumors. For the analysis, the forward model that was used to generate the numerical data serves as ground truth. The ground truth is comprised of a set of images: the real component, the imaginary component, and the magnitude component. This set is used for the reference images. The test images are comprised of the real, imaginary, and magnitude components of the reconstructed results. The reference and test images that are input to the workflow are shown in Figure 1.
Figure 1. Forward model of heterogeneously dense breast used to generate the numerical data is used as the reference images (top row). Test images (bottom row) are reconstructed from numerical data generated from the forward model.
The reference and test images shown in Figure 1 are input to a workflow represented by Figure 2. The workflow is comprised of image pre-processing, segmentation, and analysis steps that are successively applied to the test-reference pair. In the pre-processing step, the real component of the reference image is partitioned into immersion medium, breast interior, and a region-of-interest. The region-of-interest mask generated from this preliminary step is applied to the reference and test images. The region that is extracted is partitioned further into fatty, transition, fibroglandular, and tumor regions. The region-of-interest within the reference image is partitioned with a thresholding segmentation technique; an unsupervised machine learning technique is applied to the region-of-interest within the test images. The segmentation steps are described in more detail in sections 2.1. A qualitative analysis of the segmentation results is presented in section 2.2. The analysis steps of interfaces, geometric properties, and dielectric properties of the test masks are described in sections 2.3, 2.4, and 2.5, respectively.
Figure 2a. Segmentation and tissue region analysis workflow.
Figure 2b. Workflow applied to test-reference pair. Images are segmented into regions dominated by a tissue type to construct masks. Masks are used to measure discrepancies between regions within test and reference images [11].
The first step of the workflow is to partition the real component of the reference image into immersion medium, breast interior, and a region-of-interest. A closed curve delineating the skin surface from the immersion medium is formed. A mask of the breast interior is constructed from the region bound by the curve, and the region outside the breast interior is represented by a mask of the immersion medium.
Next, a region-on-interest (ROI) mask is constructed by uniformly contracting the breast interior mask by some amount (e.g., 3.5 mm) using the morphological contraction method described in [13, 14]. Note that the value of the contraction is adjusted by the user. A value of 3.0 mm is used for this example, but the value that is selected typically exceeds the skin thickness employed by the forward model represented in the reference image. Artifacts near the skin/fat interface of the test images are expected; therefore, this approach allows artefacts on the periphery of the breast interior to be excluded from analysis of interior structures.
It is not necessary to repeat the process to construct masks for the imaginary and magnitude components of the reference image, as the masks are equivalent. That is, only one set of masks for the immersion medium, breast interior, and region-of-interest is constructed. Furthermore, the reference and test images share the same immersion medium, breast interior, and region-of-interest masks. For both the reference and test images, the immersion medium and skin are considered as background; only the portion of the breast that is within the region-of-interest (i.e., interior to the skin) is partitioned into tissue types, and is discussed in the following sections.
Figure 3. Tissue masks of the real component of the reference image (left column) and each component of the test images (three right-most columns).
2.1.2 Segmentation of region-of-interest within reference images and construction of tissue type image
The region-of-interest within the real component of the reference images is delineated into group 3, group 2, group 1, and malignant tissues which we refer to as fatty, transition, fibroglandular, and tumor, respectively. The regions containing these tissues are segmented using a simple thresholding segmentation procedure described in [12] (section IV.A). The rationale for using this approach is that the forward models are constructed using a methodology that maps pixel values of the magnetic resonance (MR) images to a corresponding tissue type. The tissue types, in turn, assume values consistent with studies described in [15] and [16]. Fatty tissue is partitioned from the reference first by calculating the relative permittivity of the Debye model corresponding to group 3 (75 percentile) at the incident field frequency as documented in [15]. This value establishes the threshold to segment the fatty tissue within the breast interior. Namely, all pixel values less than the threshold value within the region-of-interest are classified as fatty tissue. The remaining tissue types are identified and segmented in a similar manner.
The segmentation process results in the partitioning of the region-of-interest into the following tissues: fatty, transition, fibroglandular, and a tumor. The segmented regions are, in turn, used to form binary images referred to as masks. The reference masks are denoted as refmask. The interpretation of these binary images is described in section 1.1. Only the real part of the reference images is segmented to form masks, as the imaginary and magnitude results are equivalent. Example reference masks (refmask) constructed from the reference image using this technique are shown in the far left column of Figure 3. Note that the transition and fibroglandular tissue masks are combined to form a single glandular mask. The transition and fibroglandular masks are used to construct a tissue type image. The construction of the tissue type image is described next.
The image is constructed by mapping each mask to a tissue type. When using the threshold technique, each tissue type is assumed known based on the threshold value applied. Therefore, mapping the segmented masks to the corresponding tissue type is straightforward. Namely, the mask corresponding to fatty tissue is mapped to the fatty label, the transition mask is mapped to the transition label, and so on. The region outside the region-of-interest is mapped to background. The union of the disjoint tissue types with their associated labels leads to an image shown in the left panel of the middle row of Figure 4.
Figure 4. Heterogeneously dense breast reference and test images (top-row); Tissue type images (middle-row); Cluster map of tissues within test images for final iteration of segmentation algorithm (bottom-row).
Figure 5. Flow diagram of segmentation algorithm used to refine partitioning of breast interior [17].
An automatic segmentation technique is applied successively to the real, imaginary, and magnitude components of the test images. It is comprised of an iterative unsupervised machine learning approach and is described in [17,18]. First, the region-of-interest mask is applied to the real component of the test images and the automatic segmentation algorithm applies the k-means clustering algorithm to the extracted region in an iterative fashion. After each iteration, a statistical test evaluates the hypothesis that the cumulative distribution function of the pixel/voxel values within clusters at the previous and current iterations originate from different distributions. If the hypothesis does not fail, then the number of clusters used by the algorithm is incremented, otherwise convergence is implied and the image is not partitioned further. The iterative procedure is summarized by the flow diagram presented in Figure 5.
The imaginary component of the test images is then selected for segmentation and the process is repeated. The process continues until all components of the test image set have been segmented. For this example, when the segmentation algorithm is terminated, the images of the real, imaginary, and magnitude component are partitioned into 10, 9, and 9 clusters, respectively. The cluster maps for the final iteration of the segmentation algorithm are shown in the bottom row of Figure 4. Note that a thresholding technique was applied to the region-of-interest of the real component of the reference image, so a cluster map is not created for this image.
Next, a mapping function assigns the clusters to binary images that have the same dimensions as the test images and are referred to as masks. The function is described in detail in [17], so only a summary is presented. Given the region-of-interest has been partitioned into
Cluster
The cluster-to-mask mapping function that maps clusters,
The test image is segmented into regions that form binary images. There is a test mask, testmask, for each region partitioned. The fibroglandular and transition masks are combined to form a single glandular mask. Figure 3 shows the tissue masks constructed by the segmentation process.
Similar to the operations performed on the reference image, each segmented mask is mapped to the corresponding tissue type. For example, the mask corresponding to fatty tissue is mapped to the fatty label, the transition mask is mapped to the transition label, and so on. The region outside the region-of-interest is mapped to background. The union of the disjoint masks with their associated labels leads to a tissue type image shown in the middle row of Figure 4. The tissue type images are visualization tools used for qualitative purposes and can assist with the interpretation of the microwave tomography reconstructions. Conversely, performance metrics are applied to the tissue masks and are used for the quantitative analysis of the images.
The mapping function that assigns the clusters to binary images is described in more detail in [17] (section 2.3). Moreover, refer to the video demonstrations that demonstrate the application of the work flow to various microwave tomography images.
The left panel of Figure 6 displays an image constructed from the union of masks formed with malignant tissue segmented from Re{ϵ(r)},Im{ϵ(r)}, and |ϵ(r)|. The integer value 0–3 indicates the number of intersecting masks, where 3 represents the highest agreement case among Re{ϵ(r)},Im{ϵ(r)}, and |ϵ(r)|. Researchers may find this image, along with the images of estimated interfaces in the middle and right panels, to be useful in interpreting reconstructed images.
Figure 6. Heterogeneously dense breast qualitative image analysis. Reference image contour (red line) superimposed onto union of test image tumor masks (left), where integer value (0–3) corresponds to number of intersecting masks. Tumor mask contours (middle), and glandular mask contours (right) with contours extracted from reference image (red–line), test image Re{ϵ(r)} (blue–line), Im{ϵ(r)} (green–line), and |ϵ(r)| (black–line).
The boundary of each test mask is sampled uniformly, resulting in a set of edge points. The edge points collectively represent estimated points on each interface that delineates a tissue type. A contour is fitted to the estimated points and is superimposed onto the reference image along with the corresponding contour of the reference mask. The reference image adds context to the test and reference contours, so the image is helpful to visually evaluate the quality of the interface extracted from the data. Contours of the reference and test interfaces extracted from the glandular masks are shown in Figures 7. Likewise, interfaces extracted from the tumor masks are shown in Figures 8-10. The reference contour is shown in red while the test (or reconstructed) contour is represented with the blue line.
The estimated points sampled from the boundary of a test mask are compared with the corresponding boundary of the reference mask to measure how accurately the interface is extracted by the reconstruction algorithm. More specifically, the ability of the reconstruction algorithm to accurately extract a tissue interface is evaluated by measuring the distance each edge point on a reconstructed interface is to the nearest point on the corresponding reference surface (refer to section 1 description of metrics). A negative distance implies that the estimated point is within the reconstructed region, whereas a positive distance implies the opposite.
In the center panels of Figures 7-10, selected points of the estimated reconstructed interface along with the errors are superimposed onto the reference contour to help visualize how the errors are spatially distributed along the interface. A quantitative analysis of the errors is provided by plotting the error distribution along with an estimate of the mean and variance (or spread) of the errors. This information is shown in the right hand panels of Figures 7-10.
Figure 7. Interface analysis results of reconstructed fat-gland interfaces extracted from heterogeneously dense breast masks. Reconstructed interface is compared with only those interfaces from reference regions for which the estimated interface touches or intersects.
Figure 8. Interface analysis results of reconstructed tumor interfaces extracted from real component of test image. Reconstructed interface is compared with interface extracted from reference image.
Figure 9. Interface analysis results of reconstructed tumor interfaces extracted from imaginary component of test image.
Figure 10. Interface analysis results of reconstructed tumor interfaces extracted from magnitude component of test image.
Metrics 1-8, 16, and 17 shown in Table 1 are applied to the test and reference masks to evaluate the accuracy with which the geometric properties of the underlying structures are reconstructed. Values of some of the metrics are sensitive to the size of the glandular or malignant region relative to the entire image. Accordingly, when applying the functions, the masks are evaluated within a region of interest defined as the box that bounds the union between the reference and reconstructed masks. Each contour and associated region is compared with the nearest reference contour and region for reconstructed Re{ϵ(r)},Im{ϵ(r)}, and |ϵ(r)| components.
Like the interface analysis, the analysis accommodates cases in which there is a requirement for the reconstructed region to be compared with many small isolated reference regions scattered throughout the interior. The geometric property analysis of the glandular, and tumor regions associated with the real, imaginary, and magnitude components are shown in Figures 11 – 14. In the figures, note that the test mask is referred as the Recon. mask and the reference mask is referred as the Forward mask. Moreover, edge points extracted from the test and reference masks that are used to construct contours to estimate interfaces are shown with blue (Rec) and red (Fwd) lines, respectively.
Figure 11. Geometric analysis of glandular masks. Geometric analysis applied to each tissue mask within a bounding box by comparing mask and interface extracted from test image with corresponding mask and interface extracted from reference image.
Figure 12. Geometric analysis of tumor masks associated with real component.
Figure 13. Geometric analysis of tumor masks associated with imaginary component.
Figure 14. Geometric analysis of tumor masks associated with magnitude component.
Each reference mask is applied to the appropriate component of the reference image set to extract the region of the image that corresponds to the tissue group represented by the mask (e.g., tumor region). These segmented property values are referred to as the reference tissue, reftissue, of the region. The process is repeated for each mask, so for each refmask there is a corresponding reftissue. Likewise, the test masks are applied to the reconstructed images. These segmented property values are referred to as the test tissue, testtissue, of the region. Like the geometric analysis, the test tissue is only compared with a reference tissue if the masks used to extract the tissues overlap.
The reference and test tissues contain both geometric and dielectric property information. Metrics 18-19 shown in Table 1 are applied to these regions evaluate the accuracy with which both the geometric and dielectric properties of these underlying structures are reconstructed. This aspect of accuracy is measured with the similarity between the reference and test tissue profiles using the normalized cross-correlation function described in Description of Metrics of section 1. Distortion of the structure and the presence of artifacts are sensed by the metric and how accurately the electric properties are reconstructed within the structure.
The dielectric property analysis of the glandular, and tumor regions associated with the real, imaginary, and magnitude components are shown in Figures 15 – 18. In the figures, note that the test tissue is referred as the Inverse model and the reference tissue is referred as the Forward model.
Figure 15. Dielectric property analysis of glandular tissue. Dielectric property analysis applied to reconstructed tissue extracted from each tissue mask by comparing with corresponding reference tissue extracted from reference image.
Figure 16. Dielectric property analysis of tumor tissue associated with real component.
Figure 17. Dielectric property analysis of tumor tissue associated with imaginary component.
Figure 18. Dielectric property analysis of tumor tissue associated with real component.