Maven Plugin

The RRE Maven Plugin is the component which is in charge of executing the RRE evaluation process during a Maven build cycle. The following RRE Maven plugin are available:

• io.sease:rre-maven-solr-plugin:<version>: for Apache Solr
• io.sease:rre-maven-elasticsearch-plugin:<version>: for Elasticsearch
• io.sease:rre-maven-generic-search-plugin:1.0 for other APIs

You should replace the version placeholder with the platform version you want to use. We choose to go with the same versions of the target platforms so the association between the product and the plugin is easy to remember.
We are constantly updating the supported versions for both platform, but it's possible that your specific version couldn't yet in.

The following snippet provides an example of the (Solr in this case) plugin configuration.

<plugin>
    <groupId>io.sease</groupId>
    <artifactId>rre-maven-solr-plugin</artifactId>
    <version>7.1.0</version>
    <configuration>
        <configurations-folder>${basedir}/src/solr/configuration_sets</configurations-folder>
        <corpora-folder>${basedir}/src/solr/corpora</corpora-folder>
        <ratings-folder>${basedir}/src/solr/ratings</ratings-folder>
        <templates-folder>${basedir}/src/solr/templates</templates-folder>
        <fields>id,product_name</fields>
        <include>
            <param>v1.[0|1]</param>
            <param>v1.10</param>
        </include>
        <exclude>
            <param>v2.*</param>
        </exclude> 
        <maximumGrade>3</maximumGrade>
        <missingGrade>2</missingGrade>
        <metrics>
            <param>io.sease.rre.core.domain.metrics.impl.Precision</param>
            <param>io.sease.rre.core.domain.metrics.impl.Recall</param>
            <param>io.sease.rre.core.domain.metrics.impl.ReciprocalRank</param>
            <param>io.sease.rre.core.domain.metrics.impl.AveragePrecision</param>
            <param>io.sease.rre.core.domain.metrics.impl.NDCGAtTen</param>
            <param>io.sease.rre.core.domain.metrics.impl.PrecisionAtOne</param>
            <param>io.sease.rre.core.domain.metrics.impl.PrecisionAtTwo</param>
            <param>io.sease.rre.core.domain.metrics.impl.PrecisionAtThree</param>
            <param>io.sease.rre.core.domain.metrics.impl.PrecisionAtTen</param>
        </metrics>
        <parameterizedMetrics>
          <!-- Parameterized metric configuration - see below -->
        </parameterizedMetrics>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>evaluate</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Where:

• configurations-folder: the folder that contains the configuration sets. The content, as described here depends on the target search platform.
• corpora-folder: the folder that contains the datasets. Each datasets must have the format required by the target search platform. See here for more details.
• ratings-folder: the folder that contains one or more ratings/judgements set. See here for more details.
• templates-folder: the folder that contains the query templates.
• fields: RRE collects, for each query execution, the top 10 results. This parameter declares the stored fields that will be retained, for each result.
• include/exclude: optional constraints which declare the versions which will be part of the evaluation process. Values can be literals or regex. In the example above, we want to include version v1.0,v1.1,v1.10 and we want to exclude all versions 2.*
• maximumGrade: the maximum grade expected in a set of judgements, used by certain metrics. May be overridden in the parameterizedMetrics section for specific metric evaluations.
• missingGrade: the grade that should be assigned to documents that have not been graded.
• metrics: a list of metrics that will be computed in the evaluation process. Each metric is declared using its FQDN. See here for an updated list of all available metrics.

As you can see from the last part, you can bind the evaluation to a specific phase (package in the example above) or you can run it isolately with the following command:

> mvn rre:evaluate

Each concrete plugin implementation (e.g. ApacheSolr or Elasticsearch at the moment) can add specific configuration parameters. The following sections list those additional settings.

Elasticsearch

The Elasticsearch plugin binding allows an additional "plugins" parameter which is a list of custom plugins we want to load as part of the embedded instance. This is useful when the project contains some customisations (i.e. a plugin) in the search workflow we want to deploy in the Elasticsearch cluster.

<plugins>
    <param>com.yourcompany.project.PluginXYZ</param>
    <param>com.othercompany.es.plugins.PluginHJQ</param>
    ...
</plugins>   

Generic search

The generic search API provides the option to connect to search APIs other than Solr or Elasticsearch (for example, a web application in front of Solr, Elasticsearch, or another search platform). To take advantage of this you will need to build your own adapter from RRE to your search API, which is then used by the Generic search plugin.

Further information is available in the plugin directory in the repository.

Parameterized Metrics

Some metrics are parameterized to allow configurations other than the defaults. For example, the below parameterizedMetric f1At1 will evaluate F-Measure with the parameters beta=1 and k=1, and store the output value in the field f1@1. Similarly, the expectedReciprocalRank will evaluate ERR@10, with a maximum grade of 4, and a missing grade of 2.5, storing the output value in the field ERR@10_missing2.5:

        <metrics>
            <param>io.sease.rre.core.domain.metrics.impl.Precision</param>
            <param>io.sease.rre.core.domain.metrics.impl.Recall</param>
        </metrics>
        <parameterizedMetrics>
            <f1At1>
                <class>io.sease.rre.core.domain.metrics.impl.FMeasureAtK</class>
                <beta>1</beta>
                <k>1</k>
            </f1At1>
            <expectedReciprocalRank>
                <class>io.sease.rre.core.domain.metrics.impl.ExpectedReciprocalRank</class>
                <k>10</k>
                <maximumGrade>4</maximumGrade>
                <missingGrade>2.5</missingGrade>
                <name>ERR@10_missing2.5</name>
            </expectedReciprocalRank>
        </parameterizedMetrics>

Note: All parameterized metrics must supply a class element containing the full evaluation class name.

For a full set of additional parameters, see the Controlling evaluation section of the Evaluation Measures wiki page.