gt4ireval - Generalizability Theory for Information Retrieval Evaluation

Provides tools to measure the reliability of an Information Retrieval test collection. It allows users to estimate reliability using Generalizability Theory and map those estimates onto well-known indicators such as Kendall tau correlation or sensitivity.

Last updated 7 years ago

4.40 score 5 stars 7 scripts 451 downloads

ircor - Correlation Coefficients for Information Retrieval

Provides implementation of various correlation coefficients of common use in Information Retrieval. In particular, it includes Kendall (1970, isbn:0852641990) tau coefficient as well as tau_a and tau_b for the treatment of ties. It also includes Yilmaz et al. (2008) <doi:10.1145/1390334.1390435> tauAP correlation coefficient, and versions tauAP_a and tauAP_b developed by Urbano and Marrero (2017) <doi:10.1145/3121050.3121106> to cope with ties.

Last updated 5 years ago

3.48 score 6 stars 10 scripts 106 downloads

simIReff - Stochastic Simulation for Information Retrieval Evaluation: Effectiveness Scores

Provides tools for the stochastic simulation of effectiveness scores to mitigate data-related limitations of Information Retrieval evaluation research, as described in Urbano and Nagler (2018) <doi:10.1145/3209978.3210043>. These tools include: fitting, selection and plotting distributions to model system effectiveness, transformation towards a prespecified expected value, proxy to fitting of copula models based on these distributions, and simulation of new evaluation data from these distributions and copula models.

Last updated 6 years ago

3.00 score 20 scripts 91 downloads