A new testing process is described designed to compare conventional retrieval (MEDLARS) and automatic text analysis methods (SMART). The results obtained with a collection of documents chosen independently of either SMART or MEDLARS indicate that a simple automatic extraction of keywords from document abstracts produces a 30 to 40 percent loss compared with MEDLARS indexing. A replacement of the unranked Boolean searches used in MEDLARS by the standard ranked output normally provided by SMART reduces the loss to between 15 and 20 percent. When an automatically generated word control list or a thesaurus is used as part of the SMART analysis, the results are comparable in effectiveness to those obtained by the intellectual MEDLARS indexing. Finally, the incorporation of user feedback procedures into SMART furnishes an improvement over the normal MEDLARS output of 15 to 30 percent. One concludes again that no technical justification exists for maintaining controlled, manual indexing in operational retrieval environments.