Challenging interdisciplinary applications inspire new methodological developments in data understanding. Two somewhat disjoint communities provide current solutions to data understanding. Statistical inference approaches based on abstract models allow incorporation of physics priors and parametric uncertainty. But to provide accurate models for complicated real-world data, one is often challenged by the curse of dimensionality. Alternatively, machine learning, especially the deep learning community, provides empirical descriptions of large complicated datasets. However, little prior knowledge is incorporated in current design of deep neural networks and such methods are often challenged by problems including data scarcity and limited transferability of the models. This dissertation includes methodological development in image understanding from each of the two perspectives: (1) Using statistical inference based on analytical models, 3-D spatial structure and temporal dynamics of nanoscale particles were reconstructed directly from large sets of cryo electron microscopy data. With a statistical framework incorporating the continuous heterogeneity among the imaged particles, a generative mechanical model was developed to provide sparse and analytical parametrization of the stochastic description of particle structure. This work contributes a systematic way to incorporate a fourth (temporal) dimension to the concept of 3D reconstruction. (2) Via deep neural networks-based machine learning approaches, the problem of concept learning in computer vision was investigated. Motivated by the challenge of data scarcity, a deep generative model-based framework, ConceptGAN, was developed to decompose data into transferable and composable semantic concepts and generatively recompose physically meaningful but unseen data, without complete training data over the joint latent space. It contributes a smart data augmentation technique which provides informative augmentation to improve the resilience of real-world applications. Finally, this dissertation concludes with a discussion on potential future research directions, in particular, on how methodological ideas from both the two perspectives of physics-based modeling and of deep learning can be fused to provide hybrid solutions that incorporate the strengths of both components, especially targeting real-world challenges including resilience, robustness, transferability and interpretability of the solutions.