Please use this identifier to cite or link to this item: http://hdl.handle.net/10397/12034
PIRA download icon_1.1View/Download Full Text
Title: Extending the extreme physical information to universal cognitive models via a confident information first principle
Authors: Zhao, X
Hou, Y
Song, D
Li, W 
Issue Date: 2014
Source: Entropy, July 2014, v. 16, no. 7, p. 3670-3688
Abstract: The principle of extreme physical information (EPI) can be used to derive many known laws and distributions in theoretical physics by extremizing the physical information loss K, i.e., the difference between the observed Fisher information I and the intrinsic information bound J of the physical phenomenon being measured. However, for complex cognitive systems of high dimensionality (e.g., human language processing and image recognition), the information bound J could be excessively larger than I (J ? I), due to insufficient observation, which would lead to serious over-fitting problems in the derivation of cognitive models. Moreover, there is a lack of an established exact invariance principle that gives rise to the bound information in universal cognitive systems. This limits the direct application of EPI. To narrow down the gap between I and J, in this paper, we propose a confident-information-first (CIF) principle to lower the information bound J by preserving confident parameters and ruling out unreliable or noisy parameters in the probability density function being measured. The confidence of each parameter can be assessed by its contribution to the expected Fisher information distance between the physical phenomenon and its observations. In addition, given a specific parametric representation, this contribution can often be directly assessed by the Fisher information, which establishes a connection with the inverse variance of any unbiased estimate for the parameter via the Cram?r-Rao bound. We then consider the dimensionality reduction in the parameter spaces of binary multivariate distributions. We show that the single-layer Boltzmann machine without hidden units (SBM) can be derived using the CIF principle. An illustrative experiment is conducted to show how the CIF principle improves the density estimation performance.
Keywords: Boltzmann machine
Fisher information
Information geometry
Parametric reduction
Publisher: MDPI AG
Journal: Entropy 
ISSN: 1099-4300
DOI: 10.3390/e16073670
Rights: © 2014 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
The following publication Zhao, X., Hou, Y., Song, D., & Li, W. (2014). Extending the extreme physical information to universal cognitive models via a confident information first principle. Entropy, 16(7), (Suppl. ), 3670-3688 is available athttps://dx.doi.org/10.3390/e16073670
Appears in Collections:Journal/Magazine Article

Files in This Item:
File Description SizeFormat 
Zhao_Extreme_Physical_Information.pdf396.61 kBAdobe PDFView/Open
Open Access Information
Status open access
File Version Version of Record
Access
View full-text via PolyU eLinks SFX Query
Show full item record

Page views

105
Last Week
1
Last month
Citations as of Mar 24, 2024

Downloads

50
Citations as of Mar 24, 2024

SCOPUSTM   
Citations

2
Last Week
0
Last month
0
Citations as of Mar 28, 2024

WEB OF SCIENCETM
Citations

4
Last Week
0
Last month
0
Citations as of Mar 28, 2024

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.