Although machine learning researchers have introduced a plethora of useful constructions for learning over Euclidean space, numerous types of data found in various applications areas benefit from, if not necessitate, a non-Euclidean treatment. In this thesis I cover the need for Riemannian geometric constructs to (1) build more principled generalizations of common Euclidean operations used in geometric machine learning models as well as to (2) enable general manifold density learning in contexts that require it. Said contexts include theoretical physics, robotics, and computational biology. I will cover two of my papers that fit into (1) above, namely the ICML 2020 paper "Differentiating through the Fréchet Mean" as well as the ICML 2022 workshop paper "Riemannian Residual Neural Networks", presented at the second annual "Topology, Algebra, and Geometry in Machine Learning" workshop. I will also cover two of my papers that fit into (2) above, namely the NeurIPS 2020 paper "Neural Manifold Ordinary Differential Equations" and the NeurIPS 2021 paper "Equivariant Manifold Flows." Finally, I will briefly discuss the implications of the work done as well as potential directions for future work.