2017 •
Large Scale Problems in Practice: The effect of dimensionality on the interaction among variables
Authors:
Fabio Caraffini, Ferrante Neri, Giovanni Iacca
Abstract:
The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. This article performs a study on correlation between pairs of variables in dependence on the problem dimensionality. Two tests, based on Pearson and Spearman coefficients, have been designed and used in this work. In total, $86$ test problems ranging between 10 and 1000 variables have been studied. If the most commonly used experimental conditions are used, the correlation between pairs of variables (...)
The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link. This article performs a study on correlation between pairs of variables in dependence on the problem dimensionality. Two tests, based on Pearson and Spearman coefficients, have been designed and used in this work. In total, $86$ test problems ranging between 10 and 1000 variables have been studied. If the most commonly used experimental conditions are used, the correlation between pairs of variables appears, from the perspective of the search algorithm, to consistently decrease. This effect is not due to the fact that the dimensionality modifies the nature of the problem but is a consequence of the experimental conditions: the computational feasibility of the experiments imposes an extremely shallow search in case of high dimensions. An exponential increase of budget and population with the dimensionality is still practically impossible. Nonetheless, since real-world application may require that large scale problems are tackled despite of the limited budget, an algorithm can quickly improve upon initial guesses if it integrates the knowledge that an apparent weak correlation between pairs of variables occurs, regardless the nature of the problem. (Read More)
We have placed cookies on your device to help make this website and the services we offer better. By using this site, you agree to the use of cookies. Learn more