Large Language Models (LLM) emerge to became a powerful approach in the medical field. From summarizing clinical notes to making personalized predictions in clinical outcomes, LLM shows its strong capabilities in assisting human-centered clinical journey. Meanwhile, LLM integrations in clinics reveal vulnerabilities, including the potential for harmful and biased responses, hallucinations, unexplainable content generation, and a lack of comprehensive measurements to validate the accuracy and reliability of the generated content. My dissertation focuses on designing and measuring how these LLM approaches can address multiple facets of challenges in human-centered healthcare delivery: what are the current landscape in LLM integrations in the medical decision-making? How to integrate and deploy LLM for patient-centered shared decision-making process? How to align these LLM prediction process with the medical domain experts? In this thesis, we explore these questions from a varieties of perspectives. First we conduct a systematic review and meta-analysis to understand the existing trends and measurements of large language model integrations in cancer decision-making. Then, for the integration and interaction chapters, we explore how cancer patients and LLMs can collaborate in the shared decision-making process, and how to design LLMs to enhance patient education. Lastly, during implementation, we develop a benchmark to measure how LLM relevance estimates differ from clinical domain experts. We conclude in with open directions for future research.