In modern settings of data analysis, we may be running our algorithms on datasets that are sensitive in nature. However, classical machine learning and statistical algorithms were not designed with these risks in mind, and it has been demonstrated that they may reveal personal information. These concerns disincentivize individuals from providing their data, or even worse, encouraging intentionally providing fake data. To assuage these concerns, we import the constraint of differential privacy to the statistical inference, considered by many to be the gold standard of data privacy. This thesis aims to quantify the cost of ensuring differential privacy, i.e., understanding how much additional data is required to perform data analysis with the constraint of differential privacy. Despite the maturity of the literature on differential privacy, there is still inadequate understanding in some of the most fundamental settings. In particular, we make progress in the following problems: \begin{itemize} \item What is the sample complexity of DP hypothesis testing? \item Can we privately estimate distribution properties with a negligible cost? \item What is the fundamental limit in private distribution estimation? \item How can we design algorithms to privately estimate random graphs? \item What is the trade-off between the sample complexity and the interactivity in private hypothesis selection? \end{itemize}