2026-09-19T01:03:33Zhttps://ecommons.cornell.edu/server/oai/request

oai:ecommons.cornell.edu:1813/1207012026-05-15T17:53:44Zcom_1813_35col_1813_47

Aimuyo, Osayamen Singh, Rachee De Sa, Christopher Guidi, Giulia 2026-04-02T18:34:23Z 2026-04-02T18:34:23Z 2025-08 ProQuest Submission ID: 11955 ProQuest Publication ID: 30631704 https://hdl.handle.net/1813/120701 https://doi.org/10.7298/ybdg-yr88 17422617 63 pages Distributed Machine Learning (DML) is increasingly recognized as a communication-bound workload, with most existing work aiming to alleviate this bottleneck through communication-computation overlap. However, current approaches—largely reliant on CPU-managed operator scheduling and synchronous communication collectives—leave significant performance on the table. This thesis identifies, analyzes, and addresses the inefficiencies arising from the interplay between CPU-driven collective communication and highly parallel GPU computation. We demonstrate that the standard bulk-synchronous communication model underutilizes GPU interconnect bandwidth and is especially vulnerable to straggler-induced performance degradation. Focusing on dynamic workloads such as Mixture-of-Experts (MoE), we highlight two critical bottlenecks. First, we demonstrate how CPU-driven execution limits the exploitation of task locality and introduces artificial synchronization barriers across distributed GPU tasks, resulting in performance reduction due to straggler effects. Second, we observe payload inefficiency at the application layer, where existing collective primitives force unnecessary padding of GPU communication buffers. To overcome these limitations, we propose a model of complete GPU residency, where inter-GPU communication is integrated directly into GPU kernels. We realize this vision in FlashDMoE: a persistent, in-kernel, actor-style operating system with packet switching that enables complete operator fusion for Distributed MoE (DMoE) into a single kernel, the first of its kind. FlashDMoE features a modular, message-driven architecture that supports lockless execution across tens of thousands of GPU threads and distributed GPUs as well. We demonstrate how FlashDMoE addresses the all-to-all communication bottleneck in expert parallelism and enables high-throughput, GPU-initiated communication. Evaluated against state-of-the-art distributed MoE frameworks, FlashDMoE achieves up to 9x higher GPU utilization, 6x lower latency, 5.7x higher throughput, and 4x better overlap efficiency compared to state-of-the-art baselines—despite FlashDMoE using FP32 while baselines use FP16. en Attribution-NonCommercial-NoDerivatives 4.0 International https://creativecommons.org/licenses/by-nc-nd/4.0/ Accelerator Distributed Machine Learning GPU Machine Learning Mixture-of-Experts Operator Fusion OS INSPIRED COMPLETE KERNEL FUSION dissertation or thesis application/pdf Computer Science Cornell University Master of Science M.S., Computer Science https://hdl.handle.net/1813/59810.2 Publication Cornell Theses and Dissertations Aimuyo, Osayamen 5893a6ea-7af3-41d7-abc6-04bcd26ab5df open.access open.access <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="02e0b327-fe03-4f7a-9187-96e66dc9b78b"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>en</Language> <Title>OS INSPIRED COMPLETE KERNEL FUSION</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2025-08</PublicationDate> <DOI>https://doi.org/10.7298/ybdg-yr88</DOI> <Authors> <Author> <DisplayName>Aimuyo, Osayamen</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <OrgUnit /> </Publisher> </Publishers> <License>https://creativecommons.org/licenses/by-nc-nd/4.0/</License> <Keyword>Accelerator</Keyword> <Keyword>Distributed Machine Learning</Keyword> <Keyword>GPU</Keyword> <Keyword>Machine Learning</Keyword> <Keyword>Mixture-of-Experts</Keyword> <Keyword>Operator Fusion</Keyword> <Abstract>Distributed Machine Learning (DML) is increasingly recognized as a communication-bound workload, with most existing work aiming to alleviate this bottleneck through communication-computation overlap. However, current approaches—largely reliant on CPU-managed operator scheduling and synchronous communication collectives—leave significant performance on the table. This thesis identifies, analyzes, and addresses the inefficiencies arising from the interplay between CPU-driven collective communication and highly parallel GPU computation. We demonstrate that the standard bulk-synchronous communication model underutilizes GPU interconnect bandwidth and is especially vulnerable to straggler-induced performance degradation. Focusing on dynamic workloads such as Mixture-of-Experts (MoE), we highlight two critical bottlenecks. First, we demonstrate how CPU-driven execution limits the exploitation of task locality and introduces artificial synchronization barriers across distributed GPU tasks, resulting in performance reduction due to straggler effects. Second, we observe payload inefficiency at the application layer, where existing collective primitives force unnecessary padding of GPU communication buffers. To overcome these limitations, we propose a model of complete GPU residency, where inter-GPU communication is integrated directly into GPU kernels. We realize this vision in FlashDMoE: a persistent, in-kernel, actor-style operating system with packet switching that enables complete operator fusion for Distributed MoE (DMoE) into a single kernel, the first of its kind. FlashDMoE features a modular, message-driven architecture that supports lockless execution across tens of thousands of GPU threads and distributed GPUs as well. We demonstrate how FlashDMoE addresses the all-to-all communication bottleneck in expert parallelism and enables high-throughput, GPU-initiated communication. Evaluated against state-of-the-art distributed MoE frameworks, FlashDMoE achieves up to 9x higher GPU utilization, 6x lower latency, 5.7x higher throughput, and 4x better overlap efficiency compared to state-of-the-art baselines—despite FlashDMoE using FP32 while baselines use FP16.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>