Orchestrating Inter-Datacenter Bulk Transfers with CodedBulk

Abstract

Over the past decades, data centers are built across the globe in response to the ever-growing Internet traffic. The data centers are managed and operated by the service providers through private inter-datacenter wide area networks (WANs). Unlike WANs involving multiple service providers, these private inter-datacenter WANs exhibit a number of unique characteristics. For example, the end-hosts and the network fabric are controlled by one single provider; scheduled bulk traffic transfers terabits of data across expensive WAN links for replication and migration; also, major service providers embrace software-defined networking (SDN) in their inter-datacenter WANs for easier management and better performance. These unique characteristics enable new solutions that were either infeasible or impractical in traditional WANs. We present CodedBulk, an end-to-end system that reduces the bandwidth required for inter-datacenter bulk transfers. CodedBulk is rooted in a known technique from the information theory community -- network coding. Network coding has been known not only for its significant theoretical benefits but also for its challenging real-world implementation and adoption. CodedBulk addresses the technical implementation challenges of network coding by exploiting the unique characteristics of inter-datacenter bulk transfers. The design of CodedBulk is agnostic to the underlying transport layer, which allows smooth integration into existing infrastructure. We run our system prototype on inter-datacenter networks, and it demonstrates a significant throughput improvement resulting from the bandwidth reduction.