Rotating neutron stars have modes that are driven unstable by gravitational radiation reaction, principally the r-mode, a Rossby wave with $n=3,m=2$, and hence large gravitational radiation reaction. Here $n$ and $m$ label the Legendre functions associated with the mode. The r-mode instability is active when gravitational driving dominates viscous dissipation. It has been suggested that this instability can (1) set the largest angular frequency of rotation of accreting neutron stars and (2) significantly spin down newborn neutron stars preventing them from reaching millisecond periods. Both the maximum frequency that neutron stars can reach and the frequency to which newborn stars can be spun down to in the first few years after formation depend on the neutron star composition via viscous dissipation and neutrino cooling. The nonlinear development of the instability plays a very important role in determining how the saturation process works, and also illustrates how instabilities can saturate at low amplitudes as a consequence of nearly resonant excitation of other modes.

We model the nonlinear interactions between modes together with basic neutron star physics including viscous heating, cooling and spin evolution of the star. The nonlinear effects are included via three-mode couplings. We show that in most scenarios one triplet of modes is sufficient to stop the growth of the instability. To explore possible nonlinear behaviors we parameterize uncertain properties of neutron stars such as the superfluid transition temperature and the rate at which the star cools via neutrino emission. The average evolution of the mode amplitudes can usually be approximated by quasi-stationary states that change slowly with spin frequency and temperature and can be determined algebraically. The spin and temperature evolution follow or oscillate around trajectories along sequences of quasi-stationary states.

In the Low Mass X-ray Binary (LMXB) case (Chapter 2), after some brief initial oscillations, the modes settle into their quasi-stationary states and the quasi-steady approximation is almost exact. The star heats via viscous dissipation from the three modes and, if this heating is balanced by neutrino cooling, then the evolution will either be stable or enter a slow thermogravitational runaway on a very long timescale of $\approx 106$ years. The stable evolutions can be (1) cyclic - with a small cycle size and a typical frequency change of at most 10%, or (2) the star can evolve toward a full equilibrium state in which the accretion torque balances the gravitational radiation emission. Alternatively, if the cooling cannot balance the heating, a faster runaway occurs, the r-mode crosses several parametric instability thresholds, and more modes need to be included.

In the young neutron star case (Chapter 3), the pulsar is hot $T\sim 1010K$ and cools fast. The evolution depends on whether the neutrino cooling can be stopped by viscous heating from the three modes. In this case the evolution is more dynamic. After a short precursor, the modes oscillate around quasi-stationary states and the spin and temperature of the star oscillate around thermal equilibrium. There are three possible outcomes: the neutron star can spin down on different sides of or along the r-mode stability curve.
If the viscosity is too low to stop the cooling a runaway occurs.