QuickSilver Scalable Multicast

Abstract

Our work is motivated by a platform we are building to support a new style of distributed programming, in which users drag and drop live components into live documents, often without needing to write new code. The capability requires a multicast layer that scales in dimensions not previously explored. In particular, live documents generate large numbers of multicast groups with irregular overlap. Traditional reliable multicast protocols were conceived for a single group at a time, and multi-group configurations can trigger costly resource contention. Quicksilver Scalable Multicast (QSM) solves these problems using two kinds of mechanisms. First, we introduce several techniques to aggregate traffic when groups overlap. But we also identify a previously unnoticed linkage between memory footprint and CPU consumption, motivating a second class of techniques that minimize memory use and CPU loads. The resulting system is fast, scales well, and is stable under stress. Moreover, our techniques should be applicable in other high-performance distributed systems.