Vis Comput DOI 10.1007/s00371-010-0541-z ORIGINAL ARTICLE Image-based rendering of intersecting surfaces for dynamic comparative visualization Stef Busking �� Charl P. Botha �� Luca Ferrarini �� Julien Milles �� Frits H. Post �� Springer-Verlag 2010 Abstract Nested or intersecting surfaces are proven tech- niques for visualizing shape differences between static 3D objects (Weigle and Taylor II, IEEE Visualization, Proceed- ings, pp. 503���510, 2005). In this paper we present an image- based formulation for these techniques that extends their use to dynamic scenarios, in which surfaces can be manipulated or even deformed interactively. The formulation is based on our new layered rendering pipeline, a generic image-based approach for rendering nested surfaces based on depth peel- ing and deferred shading. We use layered rendering to enhance the intersecting sur- faces visualization. In addition to enabling interactive per- formance, our enhancements address several limitations of the original technique. Contours remove ambiguity regard- Electronic supplementary material The online version of this article (doi:10.1007/s00371-010-0541-z) contains supplementary material, which is available to authorized users. S. Busking ( ) �� C.P. Botha �� F.H. Post Data Visualization Group, Delft University of Technology, Delft, the Netherlands e-mail: s.busking@tudelft.nl C.P. Botha e-mail: c.p.botha@tudelft.nl F.H. Post e-mail: f.h.post@tudelft.nl C.P. Botha �� L. Ferrarini �� J. Milles Division of Image Processing (LKEB), Department of Radiology, Leiden University Medical Center, Leiden, the Netherlands C.P. Botha e-mail: c.p.botha@lumc.nl L. Ferrarini e-mail: l.ferrarini@lumc.nl J. Milles e-mail: j.r.milles@lumc.nl ing the shape of intersections. Local distances between the surfaces can be visualized at any point using either depth fogging or distance fields: Depth fogging is used as a cue for the distance between two surfaces in the viewing direc- tion, whereas closest-point distance measures are visualized interactively by evaluating one surface���s distance field on the other surface. Furthermore, we use these measures to define a three-way surface segmentation, which visualizes regions of growth, shrinkage, and no change of a test surface com- pared with a reference surface. Finally, we demonstrate an application of our technique in the visualization of statistical shape models. We evaluate our technique based on feedback provided by medical image analysis researchers, who are experts in working with such models. Keywords Comparative visualization �� Image-based rendering �� Surface comparison �� Nested surfaces 1 Introduction In this paper, we examine one class of solutions to the prob- lem of comparing the shapes of 3D surfaces. Comparison of data plays an important role in many areas of scientific research. Visualization can be useful to support compara- tive data analysis. The most common approach to compar- ative visualization of surfaces is a simple side-by-side dis- play (with similar viewing conditions) of the two surfaces under consideration. Such an approach relies on memory to compare details of the surfaces, and local distances between surfaces are hard to estimate. We identify the following re- quirements for an effective comparative visualization: ��� Differences should be made explicit, alerting the user to the presence and nature of all differences present.

S. Busking et al. ��� Visualization of differences should be local, showing not only the presence but also the precise location and ex- tent of all differences. In medical applications, for in- stance, local information is required for understanding differences between patients or studying changes in spe- cific biological structures over time. ��� The visualization should be able to show relevant differ- ences and hide irrelevant ones. As an example of what is meant by relevance, consider the alignment of surfaces prior to comparison. Inaccuracies in the registration process can result in misalignment of the resulting surfaces. This misalignment, however, is usually not relevant to the researcher���s problem. An ideal visualiza- tion could automatically distinguish between relevant and ir- relevant differences, and show only the former. However, the notion of relevance is a highly application-dependent prop- erty, which can generally only be decided by the researcher. The use of user-feedback and interactivity is therefore essen- tial to deal with this issue. Various applications can benefit from interactive visualization: ��� User interaction or guidance in the registration process. ��� Local registration for exploring differences between spe- cific parts of the objects under consideration, ignoring more global differences. ��� Analysis and comparison of dynamic or deformable sur- faces. In this paper we present a visualization for the com- parison of 3D surfaces. Our visualization is based on the proven intersecting surfaces technique, first introduced and evaluated by Weigle et al. [1]. Our contribution consists of three aspects: We present an alternative, image-based im- plementation of this technique, which enables interactive performance even when manipulating alignment or dealing with dynamic objects. Furthermore, we present enhance- ments designed to address specific limitations of the ex- isting technique. Finally, we present a case study, evalu- ating the suitability of intersecting surfaces and our en- hancements for the visualization of statistical shape mod- els [2]. Figure 1 shows the comparison of two segmentations of the same brain ventricle MRI scan using our technique. Such a visualization may give important information on the char- acteristics of a new segmentation algorithm. The yellow sur- face represents the baseline segmentation, while the blue surface shows a different segmentation. This means blue ar- eas and yellow glyphs represent areas not covered by the new segmentation, while yellow areas and blue glyphs rep- resent areas covered by this segmentation which are not in the baseline segmentation. Differences are remarkably sym- metric in overall shape, but the lack of coloring due to fog indicates distances are small and contain little local varia- tion. This may indicate these could be due to variations in Fig. 1 Comparative visualizations of two partial brain ventricle sur- faces segmented from MRI data. The top figures show the separate surfaces, illustrating the difficulty to locate differences without an in- tersecting surfaces approach. The basic intersecting surfaces visualiza- tion [1] (bottom left) clearly shows some symmetry in the differences. Our visualization adds intersection contours and thresholding to sup- press smaller differences. The resulting 3-way visualization (bottom right) reveals an area of considerable change the segmentation process. In Fig. 1(d) we applied a thresh- old at a distance of 1 mm, which reveals an area of con- siderable difference in the shape of the segmented ventri- cles. Our implementation of the techniques described in this paper is available as part of the open-source NQVTK library (http://nqvtk.googlecode.com/). 2 Related work In this section we first define the position of our technique in the comparative visualization process. We use this as a framework to present and discuss work related to our tech- nique, including the intersecting surfaces visualizations by Weigle et al., on which our approach is based. As a mean- ingful discussion of work related to our implementation of these techniques depends on the details thereof, such work is discussed in Sect. 4.1.

Image-based rendering of intersecting surfaces for dynamic comparative visualization 2.1 Domain matching and comparison Comparison can occur at any stage of the standard visual- ization pipeline [3]. However, the process always requires data sets to be aligned before differences can be determined and/or visualized. Many solutions exist for this domain matching step, the details of which are mostly application- dependent. In part this is because the processes of align- ing data and extracting differences cannot always be cleanly separated. For instance, a body of work exists which applies non-rigid registration to volume data [4, 5] or to features extracted from it [6]. The deformation field resulting from this process is then analyzed to determine differences. This deformation field will typically also contain irrelevant dif- ferences due to imperfect alignment, such as tissue defor- mations caused by patient movement rather than by patho- logical processes. Our technique can be applied after domain matching in order to directly visualize the remaining differences between two surfaces. This enables such differences to be studied in detail, but also gives insight into the quality of the matching itself. We aim to be independent of the choice of domain- matching technique. We only assume that a good (possibly application-specific) solution for this stage is available: i.e., one that does not remove any of the relevant differences and preferably leaves a minimum of irrelevant differences. As perfect matching (i.e., separation of relevant and irrelevant differences) is often impossible, our technique should pro- vide enough information to assess the relevance of the dif- ferences. As stated in Sect. 1, our visualization of such differences should be explicit and local. This means a suitable technique should determine and extract localized differences after the matching step rather than simply visualizing the aligned data sets. These differences should then be mapped to clear ele- ments in the resulting visualization. Many existing compar- ison techniques are implicit in that they skip this extraction step, which means the act of comparison is left to the user. 2.2 Comparative visualization of 3D surfaces Numerous measures have been proposed that can be used to express the difference (or similarity) in shape between two surfaces most notably in the area of image retrieval [7���9] and shape retrieval [10, 11]. However, most of these mea- sures, such as the commonly used Hausdorff distance, only express similarity at a global level. The local visualization of differences can yield important insights which might not be obvious from global measure- ments. However, only a few techniques have been presented for comparing shape locally [12, 13]. Local distance mea- sures often require establishing some form of correspon- dence between the surfaces (i.e., domain matching). A com- monly used method for visualizing local distance measures is to display these on one of the two surfaces using an appro- priate color map [12, 14, 15]. This has the disadvantage that only one of the surfaces is shown the shape of the second surface is not obvious. A way to overcome this is to make one or both surfaces transparent and overlay them in the visualization. Such a nested surfaces approach (used by, e.g., Tory et al. [16]) shows all context information. Similar approaches have been proposed in uncertainty visualization (see, e.g., Johnson and Sanderson [17]). However, in these visualizations the iden- tification of differences is left to the user. This is compli- cated by the fact that overlaying multiple transparent and potentially intersecting surfaces results in an image which is not always clear to a user. In particular, understanding the shapes of transparent surfaces and classifying surfaces as being either in front or behind other surfaces can be diffi- cult perceptual tasks. Attempts have been made to resolve these issues. Tex- tures are commonly applied to improve shape perception of transparent surfaces [18]. Interrante et al. [19] proposed us- ing stroke textures based on the directions of principal cur- vatures. Bair and House [20] investigated the use of several types of grid textures, and performed user studies on their effects on perception of surface shape. Bruckner et al. [21, 22] approached similar problems in volume rendering us- ing illustrative techniques. They proposed using interactive control over a special opacity function that allows a user to focus on specific objects within a volume while also keeping contextual surfaces in view. Weigle et al. [1, 23] proposed the use of constructive solid geometry (CSG) operations to solve the perceptual problem of inside/outside classification. Their intersecting surfaces technique, described in detail in the next section, forms the basis for the visualization presented in this paper. User studies performed by Weigle et al. have shown these visualizations to be effective for the comparison and under- standing of surface shapes. 2.3 Base visualization Because of its proven effectiveness, we use the intersecting surfaces technique by Weigle et al. as the basis for our vi- sualization. The technique uses CSG operations to extract differences between two surfaces: The intersection of the closed objects formed by the surfaces represents the vol- ume in common between both objects, and is rendered as an opaque object. The remaining parts of the two surfaces represent differences, and are rendered transparently in or- der to show the intersection behind them. This solves the inside/outside classification problem, as parts of each sur- face inside the other are always opaque and outside parts are always transparent. As suggested by Interrante [19], Weigle���s visualization includes curvature-aligned glyphs on the transparent parts