Towards Dynamic Partitioning for MeasurementSet Data Format

Identifier:  TFG:9597

Handle: https://hdl.handle.net/20.500.11797/TFG9597

Authors:  Iacoponelli, Octavio

Abstract:
This thesis presents the viability research, design, and implementation of a dynamic processing plugin for astronomical MeasurementSets (.ms), enabling efficient partitioning and parallel execution. Initially developed independently, the plugin was later integrated into Dataplug and applied within the EXTRACT project. It allows cloud-native workflows to process large datasets efficiently, reducing bottlenecks in ingestion and calibration. Deployed in a Kubernetes execution backend, it achieves high speedups and demonstrates that modular, scalable pre-processing of astronomical data is both feasible and impactful for next-generation observatories.