Image and Video Abstraction by Multi-scale Anisotropic Kuwahara Filtering
Jan Eric Kyprianidis
Hasso-Plattner-Institut, Germany 
(a) Original image (b) Anisotropic Kuwahara filter (c) Proposed method
Figure 1: Example comparing the proposed multi-scale approach with the single-scale approach.
Abstract
The anisotropic Kuwahara filter is an edge-preserving filter that is
especially useful for creating stylized abstractions from images or
videos. It is based on a generalization of the Kuwahara filter that
is adapted to the local structure of image features. In this work,
two limitations of the anisotropic Kuwahara filter are addressed.
First, it is shown that by adding thresholding to the weighting term
computation of the sectors, artifacts are avoided and smooth results
in noise-corrupted regions are achieved. Second, a multi-scale com-
putation scheme is proposed that simultaneously propagates local
orientation estimates and filtering results up a low-pass filtered pyra-
mid. This allows for a strong abstraction effect and avoids artifacts
in large low-contrast regions. The propagation is controlled by the
local variances and anisotropies that are derived during the computa-
tion without extra overhead, resulting in a highly efficient scheme
that is particularly suitable for real-time processing on a GPU.
CR Categories: I.3.3 [Computing Graphics]: Picture/Image
Generation—Display algorithms; I.4.3 [Computing Methodologies]:
Image Processing And Computer Vision—Enhancement Filtering
Keywords: Non-photorealistic rendering, image abstraction,
anisotropic Kuwahara filter
1 Introduction
A common approach to creating non-photorealistic depictions is to
transform an image or video using an interactive or automatic tech-
nique. A classical example is the painting system by Haeberli [1990],
http://www.hpi3d.de
where properties such as color, size and orientation of interactively
placed brush strokes are guided by an input image. An example of
an automatic system for transforming videos using various paint-
ing styles is [Hays and Essa 2004], wherein the brush strokes are
placed automatically. Temporal coherent results are achieved by
using optical flow analysis.
Instead of focusing on simulating a particular artistic technique or
style, image abstraction refers to the process of simplifying scene
information by removing unnecessary information that is irrelevant
for a particular purpose. A common approach to image abstrac-
tion is segmentation. Several methods based on mean shift have
been proposed for abstracting images [DeCarlo and Santella 2002;
Wen et al. 2006] and videos [Wang et al. 2004; Collomosse et al.
2005]. Typically, the segmented regions created by mean shift have
rough boundaries and therefore require elaborate post-processing.
The methods that deal with video are also complicated, since they
perform processing in the spatiotemporal domain.
Another way to perform stylization and abstraction of images and
videos is through the use of edge-preserving smoothing and enhance-
ment filters. Prominent techniques in this area have in common that
they remove detail in low-contrast regions without filtering across
discontinuities, thus leaving the overall structure of the input image
unaffected. Popular examples are the bilateral filter, the Kuwahara
filter, and techniques based on or motivated by partial differential
equations (PDE).
Among these, the anisotropic Kuwahara filter [Kyprianidis et al.
2009] is of particular interest in image and video abstraction. It cre-
ates a feature-preserving and direction-enhancing look and, unlike
other nonlinear smoothing filters, is very robust against high-contrast
noise. In addition, it avoids overblurring in low-contrast regions and
provides a consistent level of abstraction across the image. More-
over, excellent temporal coherence is achieved when applied to video
on a frame-by-frame basis. However, the level of abstraction that
is achievable with the anisotropic Kuwahara filter is limited by the
filter radius. Simply increasing the filter radius is typically not a
solution, as it often results in artifacts. A possibility would be to
control the radius adaptively per pixel depending on the local neigh-
borhood, but the computational cost would be very high, as the filter

Local Orientation
and Anisotroy
Estimation
Anisotropic
Kuwahara
Filter
Merge with
smoothed
structure tensor
from previous level
Merge with result
of anisotropic
Kuwahara filter
from previous level
Upsample
to next level
Upsample
to next level
OutputInput
Build low-pass filtered pyramid
Iterate over all levels of the pyramid in coarse to fine order
Figure 2: Schematic overview of the proposed technique.
depends quadratically on the radius. In this work, a better solution is
provided by generalizing the anisotropic Kuwahara filter to operate
on multiple scales. The computations are carried out on an image
pyramid, where processing is performed in a coarse-to-fine manner,
with intermediate results being propagated up the pyramid.
2 Related Work
The use of image pyramids is a popular tool in computer graphics
and image processing. It goes back to the early work of Burt and
Adelson [1983] and Williams [1983]. While image pyramids are
often used to speed up computationally expensive operations on
large images, scale-space theory [Lindeberg 1996] provides a so-
phisticated theory for representing and analyzing images at different
scales. Several techniques in the field of non-photorealistic ren-
dering make use of image pyramids or scale-space techniques. An
example is the stroke-based painting technique by Hertzmann [1998].
Here, the final image is iteratively painted in a coarse-to-fine manner.
Starting with a large brush size, the brush size is lowered for every
iteration. Location, orientation, and color are determined by analyz-
ing the source image at a scale related to the current brush size. An
example for a technique that uses an image pyramid is the image
abstraction technique of DeCarlo and Santella [2002]. Mean-shift
segmentation is performed on each pyramid level and the results are
then organized in a tree structure representing the relationships of
the different color regions and boundaries. Guided by eye-tracking
data, this structure is then used to highlight or abstract different parts
of the image.
A well-known edge-preserving smoothing filter is the bilateral filter
[Tomasi and Manduchi 1998]. Winnemöller et al. [2006] combined
bilateral filtering with color quantization and difference of Gaussians
edges, to create cartoon-style abstractions from images and videos.
Kyprianidis and Döllner [2008] extended this approach and pre-
sented separable implementations of the bilateral and difference of
Gaussians filters that are aligned to the local image structure. Kang
et al. [2009] presented a similar system, wherein the filter shapes
of the bilateral and difference of Gaussians filters are deformed to
follow a vector field derived from the salient image features. Since
methods based on the bilateral filter preserve high-contrast edges,
they generally fail for high-contrast images where either no abstrac-
tion is performed or too much detail is removed, typically resulting
in an inconsistent abstraction.
Another popular edge-preserving smoothing filter is the Kuwahara
filter [Kuwahara et al. 1976]. The general idea behind this filter is to
divide the filter kernel into four rectangular subregions that overlap
by one pixel. The filter response is then defined by the mean of a
subregion with minimum variance. The Kuwahara filter produces
clearly noticeable artifacts, which are due to the use of rectangular
subregions. In addition, the subregion selection process is unstable if
noise is present or if subregions have the same variance, which then
results in randomly chosen subregions and corresponding artifacts.
A more detailed discussion of limitations of the Kuwahara filter can
be found in [Papari et al. 2007].
Several attempts have been made to address the limitations of the
Kuwahara filter. Papari et al. [2007] defined a new criterion for
overcoming the limitations of the unstable subregion selection pro-
cess. Instead of selecting a single subregion, the result is defined
as the weighted sum of the means of the subregions. The weights
are defined based on the variances of the subregions, resulting in
smoother region boundaries and fewer artifacts. To improve this
further, the rectangular subregions are replaced by smooth weighting
functions defined over sectors of a disc.
The anisotropic Kuwahara filter [Kyprianidis et al. 2009] builds upon
the generalized Kuwahara filtering concept by Papari et al. [2007]
and replaces the weighting functions defined over sectors of a disc
by weighting functions defined over ellipses. By adapting shape,
scale and orientation of these ellipses to the local structure of the
input, artifacts are avoided. With this adaption, directional image
features are better preserved and emphasized, resulting in overall
sharper edges and the enhancement of directional image features. A
further modification has been presented in [Kyprianidis et al. 2010b],
wherein new weighting functions based on polynomials are defined
that can be evaluated directly during the filtering process.
A further image abstraction technique has been presented by Kang
and Lee [2008]. Their approach is based on mean curvature flow in
conjunction with shock filtering. Methods based on edge-preserving
filters, such as the bilateral or the Kuwahara filter, smooth irrelevant
color variations while protecting region boundaries, but they do not
simplify the shape of those boundaries. In contrast, mean curvature
flow simplifies isophote curves and regularizes their geometry. Since
mean curvature flow does not properly protect directional image
features, Kang and Lee constrained the mean curvature flow. Mean
curvature and its constrained variant contract isophote curves to
points [Grayson 1987]. For this reason, important image features
must be protected by a user-defined mask. A further limitation is
that the technique is not stable against small changes in the input,
and therefore is not suitable for per-frame video processing. Another
technique based on diffusion and shock filtering has been recently
presented by Kyprianidis and Kang [2011], wherein flow-guided
smoothing and sharpening orthogonal to the flow are combined.
Instead of modeling the process by a PDE, approximations that
operate as local filters on a neighborhood of a pixel are used. This
makes the technique more stable and, in particular, suitable for per-
frame video processing. Interestingly, there is a connection between
PDE-based techniques and the Kuwahara filter. As shown by van