Robots operating in unstructured environments inevitably encounter failures, especially in robot caregiving scenarios. For instance, in the domain of robot-assisted bite acquisition, failures can occur because of the variation in the shapes, compliance, sizes, and textures of food items. Because of this variation, existing fully autonomous learning strategies will often fail to generalize to new environments, leading to failures at multiple stages of the robot decision-making pipeline, including perception, planning, and control. While autonomous strategies for failure recovery exist, these tend to be sample inefficient, requiring potentially unsafe interactions with the environment, or require manual human engineering. Human-in-the-loop methods for failure recovery leverage the presence of the human user, who is present during both training and deployment time, to recover from failures. However, excessive or poorly targeted queries may impose unnecessary cognitive and physical workload on the human partner. In this thesis, we develop human-in-the-loop methods for robot failure recovery that incorporate human feedback into the robot decision-making pipeline to recover efficiently from robot failures, while respecting the querying workload imposed on the human. These methods leverage confidence scores along with predictive models of human workload to decide what and when to ask for help. We first develop a human-in-the-loop method for low-level action failure recovery, LinUCB-QG, which decides when to ask for help in a contextual bandit bite acquisition setting. This method leverages a predictive model of human workload, trained on data that we collected in an online user study involving participants both with and without mobility limitations. We then extend the insights from this method to develop a human-in-the-loop framework for modular failure recovery, addressing failure modes across the pipeline. We evaluate both methods through extensive simulations, across both food datasets and controlled synthetic environments, as well as through real-robot user studies, including in-lab studies with users without mobility limitations, and in-home studies with users with mobility limitations. Through our evaluations, we find that our approaches improve recovery success while reducing the workload imposed on users. Additionally, our evaluations highlight how explicitly reasoning about both robot uncertainty and human effort can enable more efficient and user-centered failure recovery in collaborative robots.