Generalized Information Bottleneck

Abstract

Data compression is a widely used technique to reduce the transmission rate of a source signal due to the capacity bottleneck of an information channel. Classical rate-distortion theory and information bottleneck method lay the theoretical foundations for lossy data compression from an information-theoretic perspective. However, in reality there are a wide variety of lossy compression applications constrained by a non information-theoretic bottleneck, such as privacy, quantization and dimensionality bottleneck, and are hence not within the scope of the classical rate-distortion theory and information bottleneck method. Moreover, nowadays more and more applications deal with sophisticated machine learning (ML) tasks such as inference, perception and control, whose performance metric may not be standard distortion or information-theoretic objective. Therefore, in this dissertation we provide a general formulation for the family of these lossy compression problems, which we refer to as generalized information bottleneck problem. We then summarize the solutions of some existing lossy compression problems that belong to our formulation. Next, by discussing an instance which we call task-aware principal component analysis (PCA), we demonstrate that a task-aware approach, which directly aims to improve the task performance, is more desirable than a task-agnostic approach, which faithfully minimizes data reconstruction loss or uses standard method to satisfy the information bottleneck constraint regardless of the ultimate task. Furthermore, we make progress in three instances of the generalized information bottleneck problem that have not been sufficiently studied. First, we notice that mutual information and local differential privacy (LDP) are two related information bottlenecks, and we hence study the problem of mutual information maximization under LDP bottleneck. Second, we consider a task-aware LDP preservation problem that minimizes the task-specific loss for a few ML tasks under LDP bottleneck. Third, we discuss a task-aware cooperative networked control problem that minimizes the control cost of a control system with exogenous inputs under dimensionality bottleneck. In the end, we extend the generalized information bottleneck problem to a distributed setting with multiple sources and destinations by considering a task-aware network coding problem. We specifically discuss a task-aware linear network coding problem over butterfly network, which provides a preliminary step towards addressing the generalized information bottleneck problem in more general distributed settings.