We demonstrate the benefits of probabilistic representations due to their expressiveness which allows for flexible representations, their ability of capture uncertainty, and their interpretable geometric structures that are suitable for modeling hierarchical data. We show that multimodal densities can be effectively used to represent words in natural text, capturing possibly multiple meanings and their nuances. Probability densities also have natural geometric structures which can be used to represent hierarchies among entities through the concept of encapsulation; that is, dispersed distributions are generic entities that encompass more specific ones. We show an effective approach to train such density embeddings by penalizing order violations which are defined through on asymmetric divergences of probability densities.