Abstract
Here, we study two different approaches to estimate the multi-information of natural images. In both cases, we begin with a whitening step. Then, in the first approach, we use a hierarchical multi-layer ICA model [1] which is an efficient variant of projection pursuit density estimation. Projection pursuit [2] is a nonparametric density estimation technique with universal approximation properties. That is, it can be proven to converge to the true distribution in the limit of infinite amount of data and layers. For the second approach, we suggest a new model which consists of two layers only and has much less degrees of freedom than the multi-layer ICA model. In the first layer we apply symmetric whitening followed by radial Gaussianization [3,4] which transforms the norm of the image patches such that the distribution over the norm of the image patches matches the radial distribution of a multivariate Gaussian. In the next step, we apply ICA. The first step can be considered as a contrast gain control mechanism and the second one yields edge filters similar to those in primary visual cortex.
By evaluating quantitatively the redundancy reduction achieved with the two approaches, we find that the second procedure fits the distribution significantly better than the first one. On the van Hateren data set (400.000 image patches of size 12x12) with log-intensity scale, the redundancy reduction in the multi-layer ICA model yields 0.162,0.081,0.034,0.021,0.013,0.009,0.006,0.004,0.003,0.002 bits/pixel after the first, second, third, fourth, …, tenth layer, respectively.( For the training set size used, the performance decreases after the tenth layer). In contrast, we find a redundancy reduction of 0.342 bits/pixel after the first layer and 0.073 bits/pixel after the second layer for the second approach. In conclusion, the universal approximation property of the deep hierarchical architecture in the first approach does not pay off for the task of density estimation in case of natural images.
