Graphormer Boosted: Molecular Property Prediction With Enhanced Graph Spatial and Edge Encodings

Identifier:  imarina:9463655

Handle: https://hdl.handle.net/20.500.11797/imarina9463655

Authors:  Fadlallah, SA; Serratosa, F; Julià, C

Abstract:
Transformer-based architectures have gained popularity across various domains, including graph representation learning. However, selecting an optimal transformer configuration remains challenging, as attention-based models are highly sensitive to parameter choices and input value ranges. Even state-of-the-art architectures can underperform without proper tuning, while simple yet thoughtful modifications can unlock significant performance gains by better leveraging graph structures. In this work, we propose an enhancement to the Graphormer architecture that refines the attention mechanism by unifying spatial encodings, edge encodings, and similarity matrices. Specifically, we introduce nonlinear transformations, such as softmax, sigmoid, or tanh, to normalize these encodings within the attention calculation. This ensures balanced contributions from all terms, mitigating drawbacks caused by disparate value ranges. We explore two main approaches: 1) element-wise application of nonlinearities to bound spatial and edge encodings and 2) row-wise softmax normalization to emphasize their relative importance within the graph structure. The latter preserves relational information, enhancing the expressiveness of the model and improving the prioritization of node and edge. Experiments on molecular datasets demonstrate consistent performance improvements over the baseline Graphormer, highlighting the effectiveness of our approach. Additionally, our model outperforms traditional graph learning models. Our findings suggest that carefully designed nonlinear transformations over structural encodings significantly boost transformer-based graph models, offering a simple yet powerful strategy for improved graph representation learning.