We show that any factor structure for stock returns can be naturally translated into a factor structure for return volatility. We use this structure to propose a methodology for estimating forward-looking variances and covariances of both factors and individual assets from option prices at a high frequency. We implement the model empirically and show that our forward-looking volatility estimates provide useful predictions of rare disasters for both factors and individual stocks.