Multi-tenant Pub/Sub processing for real-time data streams
Visualitza/Obre
10.1007/978-3-030-10549-5_20
Inclou dades d'ús des de 2022
Cita com:
hdl:2117/129338
Tipus de documentComunicació de congrés
Data publicació2018-12
EditorSpringer
Condicions d'accésAccés obert
Tots els drets reservats. Aquesta obra està protegida pels drets de propietat intel·lectual i
industrial corresponents. Sense perjudici de les exempcions legals existents, queda prohibida la seva
reproducció, distribució, comunicació pública o transformació sense l'autorització del titular dels drets
ProjecteHi-EST - Holistic Integration of Emerging Supercomputing Technologies (EC-H2020-639595)
COMPUTACION DE ALTAS PRESTACIONES VII (MINECO-TIN2015-65316-P)
COMPUTACION DE ALTAS PRESTACIONES VII (MINECO-TIN2015-65316-P)
Abstract
Devices and sensors generate streams of data across a diversity of locations and protocols. That data usually reaches a central platform that is used to store and process the streams. Processing can be done in real time, with transformations and enrichment happening on-the-fly, but it can also happen after data is stored and organized in repositories. In the former case, stream processing technologies are required to operate on the data; in the latter batch analytics and queries are of common use.
This paper introduces a runtime to dynamically construct data stream processing topologies based on user-supplied code. These dynamic topologies are built on-the-fly using a data subscription model defined by the applications that consume data. Each user-defined processing unit is called a Service Object. Every Service Object consumes input data streams and may produce output streams that others can consume. The subscription-based programing model enables multiple users to deploy their own data-processing services. The runtime does the dynamic forwarding of data and execution of Service Objects from different users. Data streams can originate in real-world devices or they can be the outputs of Service Objects.
The runtime leverages Apache STORM for parallel data processing, that combined with dynamic user-code injection provides multi-tenant stream processing topologies. In this work we describe the runtime, its features and implementation details, as well as we include a performance evaluation of some of its core components.
CitacióVillalba, Á.; Carrera, D. Multi-tenant Pub/Sub Processing for Real-Time Data Streams. A: "Euro-Par 2018: Parallel Processing Workshops: Euro-Par 2018 International Workshops, Turin, Italy, August 27-28, 2018: revised selected papers". Springer, 2018, p. 251-526.
ISBN978-3-030-10548-8
Versió de l'editorhttps://link.springer.com/chapter/10.1007/978-3-030-10549-5_20
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
Multi-tenant Pub Sub Processing.pdf | 588,7Kb | Visualitza/Obre |