We consider information filtering, in which we face a stream of items too voluminous to process by hand (e.g., scientific articles, blog posts, emails), and must rely on a computer system to automatically filter out irrelevant items. Such systems face the exploration vs. exploitation tradeoff, in which it may be beneficial to present an item despite a low probability of relevance, just to learn about future items with similar content. We first present a simple Bayesian sequential decision-making model of this problem, where there is a unit forwarding cost and an user provides immediate feedback on every item forwarded. In the simple model, we can maximize expected total relevance minus forwarding cost using dynamic programming and a decomposition that exploits special problem structure, and show structural results for the optimal policy. We then extend the model in two realistic ways, allowing the user to provide periodic reviews on a bunch of accumulated items, and considering a constrained information filtering system where the user's cost of time is unknown. With that, we develop a policy that ranks items among categories with inspired costs. The proposing methods are especially useful when facing the cold start problem, i.e., when filtering items for new users without a long history of past interactions. We then present an application of the information filtering method to the arxiv.org repository of scientific articles, and show its implementation status in my.arxiv.org, a beta testing version of the website with recommender systems.