Many multivariate analysis problems are unified under the framework of linear projections. These projections can be tailored towards the analysis of variance (principal components), classification (discriminant analysis) or network recovery (canonical correlation analysis). Traditional techniques form these projections by using all of the original variables, however in recent years there has been a lot of interest in performing variable selection. The main goal of this dissertation is to elucidate some of the fundamental issues that arise in highdimensional multivariate analysis and provide computationally efficient and theoretically sound alternatives to existing heuristic techniques