With the increasing size of modern datasets, methods for inference under classical statistical models are often infeasible due to their poor dependence on the size of the data observations. A potential way to alleviate these issues is by reducing the number of parameters associated with these classical models, often by leveraging some structure within the observations. An example of this is when data observations admit a multi-dimensional structure, wherein the tensor normal distribution has become ubiquitous in its ability to parameterize data according to the multi-way structure of the data. In particular, this allows for parameterization of the covariance along the separate modes (or 'axes') of the data observations. This thesis contributes novel advancements to the study of statistical models and geometrically informed gradient based computational methods for Bayesian inference of multi-way covariances. The first paper within this thesis connects the structure of the tensor normal's covariance with that of the classical multivariate normal. With this connection, we will discuss the corresponding Riemannian manifold geometry of tensor normal covariances, and how such structure can be leveraged for efficient MCMC sampling under an analogue of Hamiltonian Monte Carlo. In the interest of developing a more computationally convenient alternative to MCMC, our second paper concerns a flexible variational approximation to a classic multi-way covariance posterior. Within this context, the utility of our approach is demonstrated through discussions on parallelization, geodesic convexity of the evidence lower bound under our metric choice, and flexibility of representation with respect to Monte Carlo simulation. We end this thesis with a third paper, which describes a Bayesian model whose posterior has the capability of interpolating between the structure of a single Kronecker product and a more general covariance structure through the use of geometric means on the manifold of Cholesky factors. We will highlight the significance of this approach by connecting it with a well known SVD like analogue on Kronecker product spaces.