Principles on Learning New Features for Effective Dense Matching         

Feihu Zhang     Benjamin W. Wah

Abstract

In dense Matching (including stereo matching and optical flow), nearly all existing approaches are based on simple features, such as gray or RGB color and gradient or simple transformations like census, to calculate matching costs. These features do not perform well in complex scenes that may involve radiometric changes, noises, overexposure and/or textureless regions. Various problems may appear, such as wrong matching at the pixel or region level, flattening/breaking of edges and/or even entire structural collapse. In this paper, we propose two fundamental principles based on the consistency and distinctiveness of features. We show that almost all existing problems in dense matching are caused by features that violate one or both of these principles. To systematically learn good features for dense matching, we develop a general multi-objective optimization based on these two principles and apply convolutional neural networks (CNNs) to find new features that lie on the Pareto frontier. By using two-frame optical flow and stereo matching as applications, our experimental results show that the features learned can significantly improve the performance of state-of-the-art approaches. Based on the KITTI benchmarks, our method ranks first on the two stereo benchmarks and is the best among existing two-frame optical flow algorithms on the flow benchmarks.

 

Downloads

``Principles on Learning New Features for Effective Dense Matching.''
  Feihu Zhang, Benjamin W. Wah.
 Submitted to IEEE Transaction on Imaging Processing (TIP).

   [paper]
  [code]

Experiments

 
 

Reference

[1] C. Rhemann, A. Hosni, M. Bleyer, C. Rother, and M. Gelautz, “Fast cost-volume filtering for visual correspondence and beyond,” in Proc. Conf. on Computer Vision and Pattern Recognition (CVPR). IEEE, 2011, pp. 3017–3024.

[2] F. Besse, C. Rother, A. Fitzgibbon, and J. Kautz, “Pmbp: Patchmatch belief propagation for correspondence field estimation,” Int’l J. of Computer Vision, vol. 110, no. 1, pp. 2–13, 2014.

[14] J. Zbontar and Y. LeCun, “Stereo matching by training a convolutional neural network to compare image patches,” J. of Machine Learning Research, vol. 17, pp. 1–32, 2016.

[18] M. Bleyer, C. Rhemann, and C. Rother, “Patchmatch stereo-stereo matching with slanted support windows.” in British Machine Vision Conference (BMVC), vol. 11, 2011, pp. 1–11.