Performance Evaluation Tools for Stereo Vision Analysis in Uncontrolled Environments

Abstract

Uncontrolled scenarios faced by vision-based outdoor applications represent a challenge for the vision algorithms. Constant changes in the recording conditions and the broad diversity of objects present in a scene might not be all considered or correctly modelled while testing the approaches in controlled environments. Highly accurate results in all kind of recording scenarios are required for delicate systems such as driver-assistance systems. To define accuracy or any other performance parameter in real-world environments, it is necessary to objectively measure the performance of the algorithms using the same type of data that would be fed into the algorithms. There is a lack of methods for evaluating the depth values estimated by using stereo-vision algorithms. Currently, evaluation is performed using limited in both quantity and diversity indoor data sets. The main obstacle for defining an outdoor evaluation scheme, is the absence of ground truth. This thesis proposes a set of tools designed to evaluate the performance of binocular stereo-vision algorithms for outdoor applications. To deal with the absence of ground truth, two approaches are suggested and discussed. The first approach, incorporates an extra camera into a binocular stereo-vision system. Then the depth data calculated by a stereo-vision algorithm is used to warp the reference image of the input stereo pair into a virtual image that registers the scene as if it would have been recorded with the extra camera. The evaluation is done by comparing the image grabbed with the extra camera and the generated virtual image. In the second approach, we generate sparse but nearly accurate stereo-ground truth data using a laser range-finder. To deal with the inherent sparse data from the range-finder, we define reference ground truth patches that lead to the evaluation of the majority of pixels in a disparity map. We presents and discuss experiments using both proposed approaches. The two techniques evaluate fairly the generated disparity maps, following patterns determined by the circumstances present in the image sequences under analysis. Evaluation can not only lead to a categorisation of existing algorithms but it can also enable the selection of an appropriate algorithm for a specific application.