We present a Bayesian framework for tracking an object in a sequence of image frames. A maximum a posteriori (MAP) recognition method is used to detect the object in each image frame, and a Kalman filter is used to estimate the true location from these observed locations. There is a natural feedback loop between the recognition method and the Kalman filter. The recognition method requires a prior on object location which is provided by the Kalman filter, and the Kalman filter requires an observed location which is provided by the recognition method. This framework has two desirable properties. First, the threshold for recognition in each frame depends on the system noise of the Kalman filter. This allows the system to identify partially occluded or distorted objects as long as the predicted locations are accurate. But requires a very good match if there is uncertainty as to the object location. Second, the search area for the recognition method is adaptively pruned using the current level of noise in the system, yielding an efficient overall method. Promising experimental results are demonstrated.