Machine learning and deep learning have recently witnessed great successes in various fields such as computer vision and natural language processing.In many image analysis applications, deep learning and neural networks have achieved state-of-the-art results, thanks to hardware advancement and improved data quality and accessibility over the past decades. Deep learning based methods such as convolutional neural networks (CNNs) and Transformer have shown better than human performance in some visual recognition tasks including medical imaging analysis. Among all the applications that machine learning algorithms show great potential in, localizing and segmenting anatomies in medical images is one of the most important, and usually the first step before any subsequent tasks such as computer-aided diagnosis. Although there are many successful deep learning methods for keypoint detection and image segmentation, a lot of them focus on natural images, which is very different from medical imaging such as CT and MRI. Indeed, there are still many challenges we face today that hamper the adoption of deep learning in the hospital and other clinical settings. In this thesis, we talk about some of those limitations deep learning has in the field of medical image analysis, and our algorithmic innovations and solutions to these challenges.