Longitudinal changes in latent recall operations during healthy and unhealthy aging were investigated in two studies. In the first study, healthy younger adults and healthy older adults received neuropsychological exams and associative recall memory tests at three different occasions (waves A, B, and C), spanning a period of roughly 1 year and 6 months. In the second study, older adults diagnosed as healthy control (HC), mild cognitive impairment (MCI), or Alzheimer’s disease (AD) had multiple clinical/cognitive assessments at specific intervals for a period of 3 years. In both studies, the recall data from each subject was analyzed with a Markov chain—the dual-retrieval model of recall—to extract measures of latent recollective recall (direct access) and reconstructive recall (reconstruction + familiarity judgment). The notion that normal age-related declines in episodic memory reflect changes in recollective retrieval was supported. In unhealthy aging, however, declines in reconstructive recall were the main marker of disease progression and the only operation able to differentiate HC subjects from MCI subjects, and MCI subjects from AD subjects. The reported findings suggest that unhealthy aging does not simply accelerate the normal aging process because it affects memory processes that are often spared in healthy aging.