Pivoted Document Length Normalization
| dc.contributor.author | Singhal, Amit | en_US |
| dc.contributor.author | Buckley, Chris | en_US |
| dc.contributor.author | Mitra, Mandar | en_US |
| dc.contributor.author | Salton, Gerard | en_US |
| dc.date.accessioned | 2007-04-23T18:05:18Z | |
| dc.date.available | 2007-04-23T18:05:18Z | |
| dc.date.issued | 1995-11 | en_US |
| dc.description.abstract | Document length normalization is an important aspect of term weight assignment in an automatic information retrieval system. In this study, we observe that a normalization scheme that retrieves documents of all lengths with similar chances as their likelihood of relevance will outperform another scheme which retrieves documents with chances very different from their likelihood of relevance. We show that the retrieval probabilities for a particular normalization method deviate systematically from the relevance probabilities across different collections. We present pivoted normalization a technique that can be used to reduce the gap between the relevance and the retrieval probabilities. Training pivoted normalization on one collection, we can successfully use it on other (new) text collections, yielding a robust, collection independent normalization technique. We use the idea of pivoting with the well known cosine normalization scheme. We point out some shortcomings of the cosine normalization function and present two new normalization functions --- pivoted unique normalization and pivoted byte size normalization. | en_US |
| dc.format.extent | 588248 bytes | |
| dc.format.extent | 560380 bytes | |
| dc.format.mimetype | application/pdf | |
| dc.format.mimetype | application/postscript | |
| dc.identifier.citation | http://techreports.library.cornell.edu:8081/Dienst/UI/1.0/Display/cul.cs/TR95-1560 | en_US |
| dc.identifier.uri | https://hdl.handle.net/1813/7217 | |
| dc.language.iso | en_US | en_US |
| dc.publisher | Cornell University | en_US |
| dc.subject | computer science | en_US |
| dc.subject | technical report | en_US |
| dc.title | Pivoted Document Length Normalization | en_US |
| dc.type | technical report | en_US |