Models of complex systems exhibit universal properties: there is a structural hierarchy of parameter importance. Where does this hierarchy come from? What do hierarchies say about model predictions, complex systems, and the way we make sense of phenomena? This thesis explores patterns in the complex, nonlinear models we construct to understand physical and social phenomena, with a focus on the structural hierarchy of parameter importance. Using information geometry, the problem of finding and explaining patterns in models and data is translated to one of finding structure in high-dimensional geometric objects, known as model manifolds (representing the space of all possible model predictions or all data). The structural hierarchy of parameter importance is turned into a geometric hierarchy of lengths and widths of these manifolds. In the first part of the thesis, we use approximation theory to connect the underlying smoothness of models to bounds on their corresponding model manifolds, explaining global hierarchical structure. Our approach results in universal bounds on model predictions for classes of smooth models, capturing global geometric features that are intrinsic to their model manifolds. We illustrate these ideas using three disparate models from three different fields: exponential decay (physics), reaction rates from an enzyme-catalysed chemical reaction (chemistry), and an epidemiology model of an infected population (biology). In the second part, we derive a new manifold learning technique called InPCA to obtain low-dimensional visualizations of the manifolds of general, probabilistic models and data that reveal properties of their corresponding manifolds. Using replicas to tune dimensionality in high-dimensional data, we consider the zero-replica limit to discover a distance metric which preserves distinguishability in high dimensions, and an embedding with superior visualization performance. We apply InPCA to several probabilistic models, including the finite two-dimensional Ising model of atomic spins, a trained convolutional neural network, and the model of cosmology which predicts the angular power spectrum of the cosmic microwave background allowing visualization of the space of model predictions (i.e. different universes). Finally, in the third part of this thesis, we use the tools of dimensional reduction combined with advanced statistical tests to analyse the results of a study in which we quantified student behaviours in the labs of an introductory calculus based physics course. Specifically, we analyzed gendered differences in participation in these labs. We followed 143 students across multiple lab periods in two pedagogically different lab types, and performed a cluster analysis to identify different categories of student behaviour. We found that in labs designed to foster collaborative group work and promote student decision making, there was a task division along gender lines with respect to laptop and equipment use (and found no such divide among students in more guided verification labs). Specifically, women handled laptops more than men and men behaved differently depending on whether they were in mixed-gender or single-gender groups. Students were not overtly assigned tasks, and the only explicit instruction from one student to another was in the form of quick, directed comments: the gendered division of tasks at the class level was not the result of overt task allocation but rather the accumulation of subtle interactions.