IR Metrics Without a Cutoff

[Hamedloghmani]

Regarding the usage of IR utility metrics without a cutoff, the summary of answers that I have found is as follows:

When information retrieval (IR) utility metrics are used without a cutoff, it typically means that the evaluation considers the entire ranked list of retrieved documents, without truncating or limiting the evaluation to a specific number of documents.
In IR evaluation, a cutoff is often used to determine the depth or position in the ranked list up to which the evaluation will be performed. For example, a cutoff at position 10 means that only the top 10 documents in the ranked list will be considered for evaluation.
However, when utility metrics are used without a cutoff, the evaluation takes into account the relevance or utility of documents across the entire ranked list, giving equal weight to all positions. This means that the metric considers the quality of the ranking across all the retrieved documents, rather than focusing on a specific subset.
Using utility metrics without a cutoff provides a more comprehensive assessment of the ranking algorithm or system by considering the relevance or utility of documents throughout the entire result set. It helps evaluate the overall effectiveness of the ranking in providing high-quality and relevant results, without emphasizing a particular cutoff point or subset of documents.

[hosseinfani]

@Hamedloghmani
thanks for the update.
this was out initial thought. Are you sure when there is no cutoff, the while list is considered? like recall becomes 1 always?

Can you put some citations or links to support your findings?

[Hamedloghmani]

@hosseinfani
Thank you for your feedback. Sure, some of the resources I have used are as follows:

• In FA*IR: A Fair Top-k Ranking Algorithm, Section 5.3, they formulated the NDCG without cutoff from 1 to k and indicated k is the size of the query.
• In libraries that provide these metrics such as scikit-learn etc. not having cutoff leads to considering the whole query
• Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press.