Foundation models like GPT have been at the core of natural language processing, allowing a variety of downstream tasks to be achieved through fine-tuning.Potentialities of foundation models in real-time signal processing domain have been of great interest recently for their outstanding performance. Limitations of directly applying existing foundation models to real-time signals are also evident, because of the fundamental difference between the real-time signals and natural languages. In this work, we proposed the framework of utilizing innovation representations as the foundation model for real-time signals.Inspired by the Wiener-Kallianpur innovation representation of nonparametric time series, we propose the strong and weak innovation autoencoder architectures and novel deep learning algorithms that extracts the canonical independent and identically distributed innovation sequence of the time series. The innovation representations are proved to be sufficient or Bayesian sufficient statistics, which ensures that any downstream tasks based on them are lossless. Their statistical simplicity also provides efficient solutions to a wide range of inference problem with statistical guarantee. We identified three major aspects critical to the reliable and economic operation of systems: anomaly detection, forecasting, and compression.In this work, we discussed the application of innovation representations to one-class anomaly detection, probabilistic forecasting and compression of real-time signals. Both theoretical results and numerical results demonstrating the efficacy of innovation representations are presented.