Spectral feature selection and its application in high dimensional gene expression studies

Many variable selection techniques have been proposed for clustering analysis of gene expression data. Motivated by spectral learning, we propose a new filtering method that uses the correlation between features and the eigenspace of sample similarity matrix as the variable selection criteria. Spectral algorithm states that a sample similarity matrix with q strongly connected components tends to have q piecewise almost constant eigenvectors representing a specific partition of the sample space. Using distance correlation metric, our proposed method, spectral correlation (Scorrelation) measures features' correlation with the top q eigenvectors of sample similarity matrix and then infers their ability in differentiating the underlying clusters of samples. Our method has been applied to large-scale gene expression datasets. Compared to other filtering methods, our method is more effective and provides better clustering results in terms of clustering error rate and the reliability of the selected features. Our framework can be easily extended to other types of datasets for addressing clustering and classification problems.