We explore memory in neural language models as it is stored in model weights after training and in model activations ("embeddings") during inference. We first describe a method for improving embeddings by incorporating surrounding documents in the context of text retrieval. We then propose a new method for inverting text embeddings and demonstrate its applicability to the outputs of general language models. Finally, we measurement of the information content of model weights, a way to characterize the total amount of information models can store.