Upgrading a high performance computing environment for massive data processing
Visualitza/Obre
Cita com:
hdl:2117/186788
Tipus de documentArticle
Data publicació2019-10-16
EditorSpringer
Condicions d'accésAccés obert
Llevat que s'hi indiqui el contrari, els
continguts d'aquesta obra estan subjectes a la llicència de Creative Commons
:
Reconeixement 4.0 Internacional
Abstract
High-performance computing (HPC) and massive data processing (Big Data) are two trends that are beginning to converge. In that process, aspects of hardware architectures, systems support and programming paradigms are being revisited from both perspectives. This paper presents our experience on this path of convergence with the proposal of a framework that addresses some of the programming issues derived from such integration. Our contribution is the development of an integrated environment that integretes (i) COMPSs, a programming framework for the development and execution of parallel applications for distributed infrastructures; (ii) Lemonade, a data mining and analysis tool; and (iii) HDFS, the most widely used distributed file system for Big Data systems. To validate our framework, we used Lemonade to create COMPSs applications that access data through HDFS, and compared them with equivalent applications built with Spark, a popular Big Data framework. The results show that the HDFS integration benefits COMPSs by simplifying data access and by rearranging data transfer, reducing execution time. The integration with Lemonade facilitates COMPSs’s use and may help its popularization in the Data Science community, by providing efficient algorithm implementations for experts from the data domain that want to develop applications with a higher level abstraction.
CitacióPonce, L. [et al.]. Upgrading a high performance computing environment for massive data processing. "Journal of Internet services and applications", 16 Octubre 2019, vol. 10, article 19, p. 1-18.
ISSN1867-4828
Versió de l'editorhttps://link.springer.com/article/10.1186/s13174-019-0118-7
Fitxers | Descripció | Mida | Format | Visualitza |
---|---|---|---|---|
Ponce2019_Artic ... gAHighPerformanceCompu.pdf | 3,384Mb | Visualitza/Obre |