On the limits of Conditional Generative Adversarial Neural Networks to reconstruct the identification of inhabitants from IoT low-resolution thermal sensors

Abstract

One of the main objectives of smart homes is to facilitate daily life by increasing user comfort, with the potential to play a key role in revolutionizing healthcare for the elderly, the disabled and people with functional limitations. To achieve this end, smart homes will have to be able to distinguish the identity of users, their location and the activities they are performing, while also being implemented in a non-invasive way that protects the privacy of these users. Computer vision is one of the main technologies included in smart homes. However, there are drawbacks to traditional cameras, given their dependence on light and privacyrelated concerns. Thermal cameras provide a solution, as they operate regardless of light conditions (e.g. at night) while respecting users’ privacy. In this work, image reconstruction and identification of inhabitants from facial images collected by low-resolution thermal sensors has been carried out by using Conditional Generative Adversarial Neural Networks (CGANs). The system has been implemented through an IoT device with raspberry Pi and dual-vision thermal and visible-spectrum sensors installed in a real smart home to automatically collect paired visible-spectrum and thermal images. Thus, different configurations of CGANs have been implemented and analyzed to achieve the following outcomes: (1) inhabitant identification (normal and masked face) with enhanced user privacy and (2) transfer from thermal to color images in the visible spectrum. Results show that the proposed CGAN achieves a recognition rate of 95% and 94% for uncovered and masked faces. This enables user identification without registering accurate facial expressions in the color image reconstruction, protecting user privacy. Furthermore, the developed system outperformed similar approaches using low-resolution datasets and has demonstrated that the accuracy of image reconstruction depends on the resolution of the input visible-spectrum images. In addition, a contribution to high-performance computing has been made by designing a CGAN that runs efficiently on multiple GPUs, achieving increased performance and response speed of the network, as well as applicability to larger problems.