Filtering methods are among the basic tools used for state estimation and control in many areas of engineering, particularly in aerospace. For some systems, the problem of state estimation remains challenging for various reasons, including highly nonlinear dynamics and measurements, as well as unusual process and measurement noise distributions. Although a wide variety of filtering methods have been developed over the past few decades, both for general purposes and specific applications, the need for developing new methods remains strong. This thesis is devoted to the development and improvement of nonlinear filtering methods and their applications to challenging problems in astrodynamics. Most filters currently in use compute the mean and covariance of the state of the system. This is sufficient to describe a Gaussian or nearly Gaussian distribution, but to describe a wider family of possible distributions, higher-order moments are required. In this thesis, we develop the Higher-Order Unscented Estimator, which accounts for skewness and kurtosis in addition to the mean and covariance. We test this filter in simulations of three nonlinear dynamical systems and find that it is more robust than other estimators in the presence of outliers in the process and measurement noise. Next, we apply the Square Root Sigma Point Filter to the problem of autonomous cross-calibration for Earth-imaging satellites. This is a novel application of nonlinear filtering. We develop a simulation framework for a constellation of Earth-imaging satellites, which includes detailed models of the satellites' dynamics and cameras, and we demonstrate the application of this method to two satellites in coplanar low-Earth orbits. Finally, we apply a Gaussian mixture sigma point filter to the problem of exoplanet orbit fitting. To avoid the singularities associated with the classical orbital elements, we introduce a new nonsingular parametrization for Keplerian orbits. With these new elements, the filter works well, and estimates of the classical elements can be obtained by simple transformations. Throughout this work, we show that nonlinear filtering methods can be successfully applied to new dynamical systems. This is achieved through careful modeling and parametrization of the system state and measurements and the development of new filtering techniques.