Machine learning (ML) is at the core of many Internet services and applications. Practitioners evaluate ML models based on the accuracy metrics, which measures the models' predictive power on unseen future data. On the other hand, as ML systems are becoming more personalized and more important in decision-making, malicious adversaries have an incentive to interfere with the ML environment for various purposes such as extracting information about sensitive training data or inducing desired behavior in models' output. However, none of these security and privacy threats are captured by accuracy and it is unclear to what extent current ML systems could go wrong. In this dissertation, we identify and quantify a number of threats to ML systems that are not measured by conventional performance metrics: (1) we consider the privacy threats at training time, where we show that adversary can supply malicious training code to force a ML model into intentionally "memorizing" sensitive training data, and later extract memorized information from the model; (2) motivated by data-protection regulations, we identify a compliance issue where personal information might be collected for training ML models without consent, and design practical auditing techniques for detecting such unauthorized data collection; (3) we study overlearning phenomenon in deep learning models where the internal representations reveal sensitive and uncorrelated information, and discuss its implications in terms of privacy leakages and compliance with regulations; and (4) we demonstrate a secure venerability in ML models for analyzing text semantic similarity, where we propose attacks for generating texts that are semantically unrelated but judged as similar by these ML models. The goal of this dissertation is to provide ML practitioners ways for measuring risks in the ML models through threat modeling. We hope that our proposed attacks could give insights for better mitigation methods, and advocate the ML community to consider all aspects rather than only accuracy when designing new learning algorithms and building new ML systems.