Repositori institucional URV
| URV's Author/s: | Arjona Perez, Aitor |
| Author, as appears in the article.: | Arjona, Aitor; Finol, Gerard; Lopez, Pedro Garcia |
| Author's mail: | aitor.arjona@estudiants.urv.cat aitor.arjona@estudiants.urv.cat |
| Author identifier: | 0000-0001-5451-4865 0000-0001-5451-4865 |
| Journal publication year: | 2023 |
| Publication Type: | Journal Publications |
| APA: | Arjona, Aitor; Finol, Gerard; Lopez, Pedro Garcia (2023). Transparent serverless execution of Python multiprocessing applications. Future Generation Computer Systems-The International Journal Of Escience, 140(), 436-449. DOI: 10.1016/j.future.2022.10.038 |
| Paper original source: | Future Generation Computer Systems-The International Journal Of Escience. 140 436-449 |
| Abstract: | Access transparency means that both local and remote resources are accessed using identical operations. With transparency, unmodified single-machine applications could run over disaggregated compute, storage, and memory resources. Hiding the complexity of distributed systems through transparency would have great benefits, like scaling-out local-parallel scientific applications over flexible disaggregated resources in the Cloud. This paper presents a performance evaluation where we assess the feasibility of access transparency over state-of-the-art Cloud disaggregated resources for Python multiprocessing applications. We have interfaced the multiprocessing module with an implementation that transparently runs processes on serverless functions and uses an in-memory data store for shared state. To evaluate transparency, we run in the Cloud four unmodified applications: Uber Research's Evolution Strategies, Baselines-AI's Proximal Policy Optimization, Pandaral.lel's dataframe, and Scikit Learn's Hyperparameter tuning. We compare execution time and scalability of the same application running over disaggregated resources using our library, with the single-machine Python multiprocessing libraries in a large VM. For equal resources, applications efficiently using message-passing abstractions achieve comparable results despite the significant overheads of remote communication. Other shared-memory intensive applications do not perform due to high remote memory latency. The results show that Python's multiprocessing library design is an enabler towards transparency: legacy applications using efficient disaggregated abstractions can transparently scale beyond VM limited resources for increased parallelism without changing the underlying code or architecture. |
| Article's DOI: | 10.1016/j.future.2022.10.038 |
| Link to the original source: | https://www.sciencedirect.com/science/article/pii/S0167739X22003612 |
| Paper version: | info:eu-repo/semantics/publishedVersion |
| licence for use: | https://creativecommons.org/licenses/by/3.0/es/ |
| Department: | Enginyeria Informàtica i Matemàtiques |
| Licence document URL: | https://repositori.urv.cat/ca/proteccio-de-dades/ |
| Thematic Areas: | Software Saúde coletiva Medicina ii Medicina i Matemática / probabilidade e estatística Interdisciplinar Hardware and architecture Engenharias iv Engenharias iii Engenharias i Comunicação e informação Computer science, theory & methods Computer networks and communications Ciências sociais aplicadas i Ciências biológicas ii Ciências biológicas i Ciência da computação |
| Keywords: | Transparency Serverless Parallel programming Multiprocessing Faas Access transparency serverless parallel programming multiprocessing faas access transparency |
| Entity: | Universitat Rovira i Virgili |
| Record's date: | 2025-02-18 |
| Search your record at: |   |
| File | Description | Format | |
|---|---|---|---|
| DocumentPrincipal | DocumentPrincipal | application/pdf |
© 2011 Universitat Rovira i Virgili