A case study of organizational evaluation of the effectiveness of full text information retrieval systems: A macro and micro perspective

Evaluating the effectiveness of full text retrieval systems can pose a major problem for organizations. One way of evaluating effectiveness is by analyzing the user's satisfaction with the information retrieval system (macro perspective). Two specific factors of the search process can also be used to evaluate effectiveness. They are searcher behavior and characteristics and retrieval effectiveness (micro perspective). Two Federal Government agencies and two private organizations were studied to determine the method they used to evaluate the effectiveness of their full tent database retrieval systems. Results of the study showed that the macro perspective was the method most used by three of the organizations. The organizations all employed some informal methods for evaluating searcher behavior. None of the organizations used precision or recall to measure retrieval effectiveness. Recommendations for further study were made based on the results of this study.