A Language for Generating Flat Data Structures
Data structures represented with pointers are typically less performant than the same data flattened into an array. Optimizations such as turning an array of structs into a struct of arrays have been known to improve performance by taking advantage of spatial locality, and writing code that operates directly on serialized data eliminates the overhead of serialization and deserialization. However, pointer representations are typically more ergonomic and intuitive to code against compared to flattened and packed data. I introduce a new domain specific language that allows users to define data types with natural pointer representations. The compiler then creates flattened versions of the data in Rust, with convenience methods that allow the user to write code with minimal changes. The language enables the user to customize the specific layout of the flattened version of each data structure while preserving the Rust interface, making it easy to experiment with layouts to find the best performance. When tested on code that operates on real genomic data, significant speed improvements were found by swapping out the pointer data structures for the flattened version, with minimal code changes needed.