International Journal of Computer Vision 61(3), 211���231, 2005 c 2005 Springer Science + Business Media, Inc. Manufactured in The Netherlands. Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods ANDR �� ES BRUHN AND JOACHIM WEICKERT Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Building 27, 66041 Saarbrucken, �� Germany bruhn@mia.uni-saarland.de weickert@mia.uni-saarland.de CHRISTOPH SCHNORR�� Computer Vision, Graphics and Pattern Recognition Group, Faculty of Mathematics and Computer Science, University of Mannheim, 68131 Mannheim, Germany schnoerr@uni-mannheim.de Received August 5, 2003 Revised April 22, 2004 Accepted April 22, 2004 First online version published in October, 2004 Abstract. Differential methods belong to the most widely used techniques for optic flow computation in image sequences. They can be classified into local methods such as the Lucas���Kanade technique or Big�� un���s structure tensor method, and into global methods such as the Horn/Schunck approach and its extensions. Often local methods are more robust under noise, while global techniques yield dense flow fields. The goal of this paper is to contribute to a better understanding and the design of novel differential methods in four ways: (i) We juxtapose the role of smoothing/regularisation processes that are required in local and global differential methods for optic flow computation. (ii) This discussion motivates us to describe and evaluate a novel method that combines important advantages of local and global approaches: It yields dense flow fields that are robust against noise. (iii) Spatiotemporal and nonlinear extensions as well as multiresolution frameworks are presented for this hybrid method. (iv) We propose a simple confidence measure for optic flow methods that minimise energy functionals. It allows to sparsify a dense flow field gradually, depending on the reliability required for the resulting flow. Comparisons with experiments from the literature demonstrate the favourable performance of the proposed methods and the confidence measure. Keywords: optic flow, differential techniques, variational methods, structure tensor, partial differential equations, confidence measures, performance evaluation 1. Introduction Ill-posedness is a problem that is present in many im- age processing and computer vision techniques: Edge detection, for example, requires the computation of im- age derivatives. This problem is ill-posed in the sense of Hadamard,1 as small perturbations in the signal may create large fluctuations in its derivatives (Yuille and Poggio, 1986). Another example consists of optic flow computation, where the ill-posedness manifests itself in the nonuniqueness due to the aperture prob- lem (Bertero et al., 1988): The data allow to compute only the optic flow component normal to image edges. Both types of ill-posedness problems appear jointly in so-called differential methods for optic flow recov- ery, where optic flow estimation is based on computing

212 Bruhn, Weickert and Schn�� orr spatial and temporal image derivatives. These tech- niques can be classified into local methods that may optimise some local energy-like expression, and global strategies which attempt to minimise a global en- ergy functional. Examples of the first category include the Lucas���Kanade method (Lucas and Kanade, 1981 Lucas, 1984) and the structure tensor approach of Big�� un and Granlund (1988) and Big�� un et al. (1991), while the second category is represented by the clas- sic method of Horn and Schunck (Horn and Schunck, 1981) and its numerous discontinuity-preserving vari- ants (Alvarez et al., 1999 Aubert et al., 1999 Black and Anandan, 1991 Cohen, 1993 Heitz and Bouthemy, 1993 Kumar et al., 1996 Nagel, 1983 Nesi, 1993 Proesmans et al., 1994 Schn�� orr, 1994 Shulman and Herv�� e, 1989 Weickert and Schn�� orr, 2001). Differential methods are rather popular: Together with phase-based methods such as (Fleet and Jepson, 1990) they belong to the techniques with the best performance (Barron et al., 1994 Galvin et al., 1998). Local methods may offer relatively high robustness under noise, but do not give dense flow fields. Global methods, on the other hand, yield flow fields with 100% density, but are experimen- tally known to be more sensitive to noise (Barron et al., 1994 Galvin et al., 1998). A typical way to overcome the ill-posedness prob- lems of differential optic flow methods consists of the use of smoothing techniques and smoothness as- sumptions: It is common to smooth the image se- quence prior to differentiation in order to remove noise and to stabilise the differentiation process. Lo- cal techniques use spatial constancy assumptions on the optic flow field in the case of the Lucas���Kanade method, and spatiotemporal constancy for the Big�� un method. Global approaches, on the other hand, sup- plement the optic flow constraint with a regularising smoothness term. Surprisingly, the actual role and the difference between these smoothing strategies, how- ever, has hardly been addressed in the literature so far. In a first step of this paper we juxtapose the role of the different smoothing steps of these methods. We shall see that each smoothing process offers certain advantages that cannot be found in other cases. Conse- quently, it would be desirable to combine the different smoothing effects of local and global methods in or- der to design novel approaches that combine the high robustness of local methods with the full density of global techniques. One of the goals of the present pa- per is to propose and analyse such an embedding of local methods into global approaches. This results in a technique that is robust under noise and gives flow fields with 100% density. Hence, there is no need for a postprocessing step where sparse data have to be interpolated. On the other hand, it has sometimes been criticised that there is no reliable confidence measure that al- lows to sparsify the result of a dense flow field such that the remaining flow is more reliable (Barron et al., 1994). In this way it would be possible to compare the real quality of dense methods with the character- istics of local, nondense approaches. In our paper we shall present such a measure. It is simple and applica- ble to the entire class of energy minimising global op- tic flow techniques. Our experimental evaluation will show that this confidence measure can give excellent results. Our paper is organised as follows. In Section 2 we discuss the role of the different smoothing pro- cesses that are involved in local and global optic flow approaches. Based on these results we propose two combined local-global (CLG) methods in Section 3, one with spatial, the other one with spatiotemporal smoothing. In Section 4 nonlinear variants of the CLG method are presented, while a suitable multiresolu- tion framework is discussed in Section 5. Our nu- merical algorithm is described in Section 6. In Sec- tion 7, we introduce a novel confidence measure for all global optic flow methods that use energy func- tionals. Section 8 is devoted to performance evalua- tions of the CLG methods and the confidence mea- sure. A summary and an outlook to future work is given in Section 9. In the Appendix, we show how the CLG principle has to be modified if one wants to replace the Lucas���Kanade method by the struc- ture tensor method of Big�� un and Granlund (1988) and Big�� un et al. (1991). 1.1. Related Work In spite of the fact that there exists a very large number of publications on motion analysis (see e.g. (Mitiche and Bouthemy, 1996 Stiller and Konrad, 1999) for reviews), there has been remarkably little work de- voted to the integration of local and global optic flow methods. Schn�� orr (Schn�� orr, 1993) sketched a frame- work for supplementing global energy functionals with multiple equations that provide local data constraints. He suggested to use the output of Gaussian filters shifted in frequency space (Fleet and Jepson, 1990) or

Lucas/Kanade Meets Horn/Schunck 213 local methods incorporating second-order derivatives (Tretiak and Pastor, 1984 Uras et al., 1988), but did not consider methods of Lucas���Kanade or Big�� un type. Our proposed technique differs from the majority of global regularisation methods by the fact that we also use spatiotemporal regularisers instead of spa- tial ones. Other work with spatiotemporal regularisers includes publications by Murray and Buxton (1987), Nagel (1990), Black and Anandan (1991), Elad and Feuer (1998), and Weickert and Schn�� orr (2001). While the noise sensitivity of local differential methods has been studied intensively in recent years (Bainbridge-Smith and Lane, 1997 Ferm�� uller et al., 2001 J�� ahne, 2001 Kearney et al., 1987 Ohta, 1996 Simoncelli et al., 1991), the noise sensitivity of global differential methods has been analysed to a signifi- cantly smaller extent. In this context, Galvin et al. (1998) have compared a number of classical methods where small amounts of Gaussian noise had been added. Their conclusion was similar to the findings of Barron et al. (1994): the global approach of Horn and Schunck is more sensitive to noise than the local Lucas���Kanade method. A preliminary shorter version of the present paper has been presented at a conference (Bruhn et al., 2002). Additional work in the current paper includes (i) the use of nonquadratic penalising functions, (ii) the ap- plication of a suitable multiresolution strategy, (iii) the proposal of a confidence measure for the entire class of global variational methods, (iv) the integration of the structure tensor approach of Big�� un and Granlund (1988) and Big�� un et al. (1991) and (v) a more extensive experimental evaluation. 2. Role of the Smoothing Processes In this section we discuss the role of smoothing tech- niques in differential optic flow methods. For simplicity we focus on spatial smoothing. All spatial smoothing strategies can easily be extended into the temporal domain. This will usually lead to improved results (Weickert and Schn�� orr, 2001). Let us consider some image sequence g(x, y, t), where (x, y) denotes the location within a rectangular image domain , and t ��� [0, T ] denotes time. It is com- mon to smooth the image sequence prior to differentia- tion (Barron et al., 1994 Kearney et al., 1987), e.g. by convolving each frame with some Gaussian K�� (x, y) of standard deviation �� : f (x, y, t) := (K�� ��� g)(x, y, t), (1) The low-pass effect of Gaussian convolution removes noise and other destabilising high frequencies. In a sub- sequent optic flow method, we may thus call �� the noise scale. Many differential methods for optic flow are based on the assumption that the grey values of image objects in subsequent frames do not change over time: f (x +u, y+v, t +1) = f (x, y, t), (2) where the displacement field (u,v) (x, y, t) is called optic flow. For small displacements, we may perform a first order Taylor expansion yielding the optic flow constraint fx u + fy v + ft = 0, (3) where subscripts denote partial derivatives. Evidently, this single equation is not sufficient to uniquely com- pute the two unknowns u and v (aperture problem): For nonvanishing image gradients, it is only possible to determine the flow component parallel to ��� f := ( fx , fy ) , i.e. normal to image edges. This so-called normal flow is given by wn = ��� ft |��� f | ��� f |��� f | . (4) Figure 1(a) depicts one frame from the famous Hamburg taxi sequence.2 We have added Gaussian noise, and in Fig. 1(b)���(d) we illustrate the impact of presmoothing the image data on the normal flow. While some moderate presmoothing improves the re- sults, great care should be taken not to apply too much presmoothing, since this would severely destroy im- portant image structure. In order to cope with the aperture problem, Lucas and Kanade (1981) and Lucas (1984) proposed to assume that the unknown optic flow vector is constant within some neighbourhood of size ��. In this case it is possible to determine the two constants u and v at some location (x, y, t) from a weighted least square fit by minimising the function ELK (u,v) := K�� ��� ( ( fx u + fy v + ft )2 ) . (5) Here the standard deviation �� of the Gaussian serves as an integration scale over which the main contribution of the least square fit is computed. A minimum (u,v) of ELK satisfies ���u ELK = 0 and ���v ELK = 0. This gives the linear system K�� ��� ( fx2 ) K�� ��� ( fx fy ) K�� ��� ( fx fy ) K�� ��� ( fy2 ) u v = ���K�� ��� ( fx ft ) ���K�� ��� ( fy ft ) (6)