Visual context is the distillation of visual experience that enables reasonable accuracy in interpreting the natural world. Context consists of all the constraints that limit possibilities in recognizing objects and surfaces, inferring their properties, and other cognitive processes. We contend that these visual constraints are statistically learned from experience with the world. We also contend that context, once well-established through repetition of similar views of an environment, can impede memory for images of that environment. Two aspects of context are fundamental to scene perception. Identity context is cooccurrence of objects and surfaces within the scene. Spatial context is how objects and surfaces are laid out in the scene, and their spatial relationships. When objects and surfaces are not within observers' normal experience, either in identity or spatial context, observers are quick to notice. Inconsistent object effects and spatial relationship violations have been shown in research with real-world scenes. Experiments presented here reproduce inconsistent object and spatial relationship violation effects using novel abstract objects, demonstrating learning from a baseline without reference to real-world objects. In delayed match to sample tasks, participants learned spatial context quickly for multiple objects within the same scene. Learned spatial contexts markedly improved sensitivity to changes in an object's location when it appeared outside the learned area. Participants also learned identity context for novel objects, but only when at least one object differed markedly in visual appearance from others in the scene. Experiments with a very well-established context, natural scenery without human or animal presence, demonstrated that context can interfere with memory for individual images from the context. In memory comparisons with 100-image sets of a variety of human-oriented content, natural scenery, 1/f noise, or white noise, performance was lower for natural scenery image sets than for high variety. However, natural scenery performance was better than for white noise and 1/f noise. Poor performance was not due to inability to discriminate when exemplars are viewed side-by-side. Participants can also remember at least four exemplars of difficult-to-discriminate images, including noise.