Reinforcement learning agents struggle to learn robot manipulation skills when the observation space includes high-dimensional inputs such as images. Leveraging privileged state when training visuomotor policies significantly reduces the dimensionality of the inputs and the required exploration for reinforcement learning (RL) agents to solve manipulation tasks. My thesis first presents a modular approach for training visuomotor policies where we first use vision foundation models to perceive the privileged state and then we train a low- dimensional manipulation policy through RL in simulation. Through this approach, we teach a robot to perform open-vocabulary picking. My thesis then investigates the setting where this privileged information is hidden from the current observation. Recent works use a teacher policy trained with privileged information to then train a student policy with the target information via imitation learning. However, this method struggles when there is information asymmetry between the teacher and the student, causing the teacher to be unrealizable for the student. We introduce two methods to overcome this challenge: (i) an imitation learning approach that adaptively determines when the student should query the teacher for corrections, and (ii) a reinforcement learning approach that selects where to initialize training for efficient exploration. We validate our methods in both simulated and real-world robotic tasks, demonstrating significant improvements over standard teacher-student baselines in training efficiency and final performance.