Scalable and efficient data stream delivery and archiving infrastructure for real-time sensing applications

Abstract

Upcoming ubiquitous technologies are expediting the advent of many real-time applications. Examples of such applications include physical world browsing, RFID-based supply chain management, city-wide road traffic monitoring, weather forecasting, and air pollution monitoring. These applications show different scales and characteristics in terms of sensing data delivery demands. They commonly demand a deep understanding on real-time data delivery from widely distributed data sources. Also, they have highly individualized and fine-grained delivery demands in terms of data and delay specifications, e.g., data value ranges of interest, spatial and temporal resolutions, and tolerable delay, etc. Due to the remarkable scale and complexity, however, existing data delivery systems cannot support such applications effectively. We present SATI (Scalable And Traffic-efficient data delivery Infrastructure), a novel Internet-based sensing data delivery infrastructure that provides a common platform for data providers and consumers. Basically, it is comprised of a collection of proxy nodes forming an overlay network, where each proxy node conducts an in-network processing and efficient data delivery. It allows applications to specify their delivery requirements with intuitive and comprehensive delivery semantics. For scalable and efficient data delivery, SATI develops a novel delivery path management scheme based on an incremental relaxation method. The scheme enables SATI to construct and maintain efficient delivery paths satisfying a large number of delivery requests of high diversity. It fully exploits the diversity of delivery demands on both data and delay requirements, thus achieving a high level of service satisfaction and efficiency at the same time. The result from a large-scale simulation shows that SATI achieves a high level of scalability and bandwidth efficiency. Real-time sensing applications also require handling high volumes of continuously arriving data and...