Experimental quality evaluation of lattice basis reduction methods for decorrelating low-dimensional integer least squares problems

Abstract

Reduction can be important to aid quickly attaining the integer least squares (ILS) estimate from noisy data. We present an improved LLL algorithm with fixed complexity by extending a parallel reduction method for positive definite quadratic forms to lattice vectors. We propose the minimum angle of a reduced basis as an alternative quality measure of orthogonality, which is intuitively more appealing to measure the extent of orthogonality of a reduced basis. Although the LLL algorithm and its variants have been widely used in practice, experimental simulations were only carried out recently and limited to the quality measures of the Hermite factor, practical running behaviors and reduced Gram-Schmidt coefficients. We conduct a large scale of experiments to comprehensively evaluate and compare five reduction methods for decorrelating ILS problems, including the LLL algorithm, its variant with deep insertions and our improved LLL algorithm with fixed complexity, based on six quality measures of reduction. We use the results of the experiments to investigate the mean running behaviors of the LLL algorithm and its variants with deep insertions and the sorted QR ordering, respectively. The improved LLL algorithm with fixed complexity is shown to perform as well as the LLL algorithm with deep insertions with respect to the quality measures on length reduction but significantly better than this LLL variant with respect to the other quality measures. In particular, our algorithm is of fixed complexity, but the LLL algorithm with deep insertions could seemingly not be terminated in polynomial time of the dimension of an ILS problem. It is shown to perform much better than the other three reduction methods with respect to all the six quality measures. More than six millions of the reduced Gram-Schmidt coefficients from each of the five reduction methods clearly show that they are not uniformly distributed but depend on the reduction algorithms used. The simulation results of the reduced Gram-Schmidt coefficients have clearly shown that our improved LLL algorithm tends to produce small reduced Gram-Schmidt coefficients near zero with a larger probability and large reduced Gram-Schmidt coefficients near both ends of 0.5 and −0.5 with a smaller probability.