Development of a voluntary computing platform to run (scientific) applications in the browser and/or Node.js

Identifier:  TFG:9473

Handle: https://hdl.handle.net/20.500.11797/TFG9473

Authors:  Robledo Kusz, Miguel

Abstract:
This project presents PyEdgeCompute, a system for the distributed execution of parallel data analysis workloads on volunteer devices. Typically, data analysis workloads are executed either on proprietary clusters or in cloud deployments. However, the cost of these resources has motivated the search for solutions that take advantage of the dispersed and heterogeneous computational capacity of volunteer devices and at the Edge. PyEdgeCompute proposes a framework for executing data analysis directly on the Edge, automatically managing dependencies and data exchange between the different stages of the computation. PyEdgeCompute leverages WebAssembly (WASM) to ensure compatibility with a wide variety of devices, including web browsers and lightweight environments, without the need for complex installations. The system allows the client to simply define tasks, automating the distribution, execution, and coordination of computations. During development, a flexible architecture has been designed that manages data flow control and intelligent task assignment according to the availability and capacity of volunteers. The first results obtained show efficient and scalable execution in heterogeneous environments, opening the door to the widespread use of volunteer computing in complex data analytics areas. This project contributes to democratizing access to distributed computing resources, reducing the dependence on expensive cloud platforms and dedicated infrastructures, and proposes a sustainable model for the use of underutilized computational resources.