Sparse tensor computations are fundamental to a wide range of domains, such as scientific computing, signal processing, graph analytics, and efficient machine learning. Sparse workloads are predominantly memory-bound, making the design of data format a critical factor in overall performance. The growing trend toward hardware specialization has led to a proliferation of custom formats tailored to specific platforms, such as GPUs, FPGAs, and processing-in-memory devices. However, existing sparse tensor compilers provide limited support for expressing these diverse formats, converting between them, and generating efficient code accordingly. Compounding this challenge, modern GPU architectures exemplified by NVIDIA Hopper incorporate specialized hardware units, such as Tensor Core Units and the Tensor Memory Accelerator, that impose strict data layout requirements and asynchronous execution paradigms. These constraints necessitate new abstractions that express customized sparse tensor formats and optimizations that diverge significantly from conventional sparse algorithms. This dissertation addresses these challenges through two complementary thrusts: a unified format abstraction with automated compilation, and the exploration of efficient sparse kernel designs on modern GPU architectures with asynchronous execution patterns. The contributions are organized as follows. First, we introduce UniSparse, an intermediate language that provides a unified abstraction for representing and customizing sparse formats. UniSparse decouples the logical representation of a sparse tensor from its physical memory layout. Formats are expressed through index maps, which define coordinate transformations and dimension orderings, together with a set of well-defined query, mutation, and layout primitives. This compositional design enables the succinct specification of an unlimited number of custom formats, such as hybrid formats adapted to input sparsity distributions, load-balanced formats, and banked formats for multi-bank memory systems. It addresses the expressiveness gap of prior frameworks that encode sparse formats with finite attribute combinations. Second, we develop a compiler built on the MLIR infrastructure that automates both sparse format conversion and compute kernel generation across heterogeneous architectures. The compiler includes a general format conversion algorithm that analyzes source and destination format encodings and emits a modular sequence of conversion operators to incrementally transform data structures and physical layouts. This approach supports converting to a broad range of formats, including those with custom index maps, indirect mapping functions, and custom memory layouts, expanding coverage beyond state-of-the-art compilers while achieving comparable or superior conversion performance. For compute kernel generation, the compiler infers data structures and physical memory layouts from format encodings, then lowers format customization and compute kernel specifications into executable code targeting Intel CPUs, NVIDIA GPUs, AMD Xilinx FPGAs, and simulated processing-in-memory (PIM) devices. Third, we explore efficient sparse matrix-matrix multiplication (SpMM) kernel designs on modern GPU architectures featuring asynchronous execution models. The UniSparse compiler-generated kernels follow synchronous execution patterns and cannot match the performance of hand-tuned kernels on modern GPU platforms. Therefore, we present AsyncSparse, a set of high-performance SpMM kernels co-designed with the asynchronous execution features of the NVIDIA Hopper (H100) GPU architecture. We use two complementary sparse formats, Block Compressed Sparse Row (BCSR) and Window Compressed Sparse Row (WCSR), as vehicles to analyze the performance implications. For structured sparsity, we optimize a warp-specialized producer-consumer pipeline overlapping TMA data transfer with WGMMA computation using the BCSR format. For irregular sparsity, we design a WCSR kernel that loads the sparse operand via TMA and splits large row-windows across thread blocks for load balancing. Our WCSR kernel outperforms prior SpMM kernels on SuiteSparse matrices (1.47x over AccSpMM, 6.24x over cuSPARSE). Our BCSR kernel achieves a combined 2.66x end-to-end speedup on Qwen2.5-7B prefill at 90% block sparsity with 64K sequence length over cuDNN/cuBLAS. Together, this dissertation advances sparse tensor computation along two axes: abstraction and specialization. UniSparse provides a unified format abstraction that enables the compiler to automate format conversion and compute kernel generation across heterogeneous platforms; however, its loop-nest execution model cannot express asynchronous execution patterns such as warp-specialized producer-consumer pipelines. AsyncSparse fills this gap by demonstrating efficient SpMM designs that exploit these patterns on recent GPU architectures, and its systematic ablation study isolates the performance trade-offs of applying asynchronous features to sparse workloads, providing a concrete template for extending format-aware compilers with asynchronous GPU backends.