Elucidating the nature of semantic representations is important for an understanding of brain information processing, learning and memory. Semantic representations are built on the initial representations of sensory signals, but previous work has suggested that these higher-level representations are qualitatively different. Here, we probe, at an algorithmic and a mechanistic level, how the mental representations of low-level sensory information and semantic information differ. To study differences at the algorithmic level, we measure perceptual similarities in psychophysics experiments, and to study them at a mechanistic level, we measure BOLD-signal activations in parallel fMRI experiments. Both sets of experiments use a common set of stimulus domains, which contain varying levels of visual and semantic information: visual textures, pictures of animals, animal names and unexplored representations of two novel intermediate domains. The mental representations for these stimulus domains are characterized independently and then compared. For the psychophysical studies, the analysis is based on a variant of multidimensional scaling; for neuroimaging data, the focus is on representational similarity analysis. Psychophysical experiments show that low-level sensory domains and the more semantic domains do differ; the latter appear to be more tree-like. The fMRI results also show distinctions between low-level and high-level domains, but not in the same way: rather than a main distinction between low-level and semantic domains, neuroimaging measures show that words are represented differently from all the non-word stimulus domains. Together, these results support the hypothesis that sensory and semantic information are represented in different ways and delineate some of these differences, but leave open the link between behavior and brain activity.