Smallest singular value and limit eigenvalue distribution of a class of non-Hermitian random matrices with statistical application

Suppose X is an N×n complex matrix whose entries are centered, independent, and identically distributed random variables with variance 1∕n and whose fourth moment is of order O(n−2). Suppose A is a deterministic matrix whose smallest and largest singular values are bounded below and above respectively, and z≠0 is a complex number. First we consider the matrix XAX∗−z, and obtain asymptotic probability bounds for its smallest singular value when N and n diverge to infinity and N∕n→γ,0<γ<∞. Then we consider the special case where A=J=[1i−j=1modn] is a circulant matrix. Using the above result, we show that the limit spectral distribution of XJX∗ exists when N∕n→γ,0<γ<∞ and describe the limit explicitly. Assuming that X represents a ℂN-valued time series which is observed over a time window of length n, the matrix XJX∗ represents the one-step sample autocovariance matrix of this time series. A whiteness test against an MA correlation model for this time series is introduced based on the above limit result. Numerical simulations show the excellent performance of this test.