Home
The Rated Ranking Evaluator (RRE) is a search quality evaluation tool that evaluates the quality of results coming from a search infrastructure.
It is a software intended to help the Search Engineer in her daily job. Are you a Search Engineer? Are you tuning/implementing/changing/configuring a search infrastructure? Do you want to have something that gives you evidence of the improvements caused by your changes? Then you're in the right place.
There's more: RRE formalises how well a search system satisfies the user information needs, at a "technical" level, combining a rich tree-like domain model with several evaluation measures, but also at a "functional" level, providing human-readable outputs that can act as deliverables for business stakeholders.
It encourages an incremental/iterative/immutable approach when developing and evolving a search system: assuming we're starting from version 1.0, when it's time to apply some relevant change to the configuration, instead of changing that version, it's better to clone it and apply those changes to the new version.
In this way, the evaluation process will happen for all available versions and RRE will provide the delta/trend between each subsequent versions, so you can immediately get a fine-grained picture about where the system is going, in terms of relevance.
The following sections highlight the most important features of the project.
RRE is not a monolithic piece of software. Rather, it has been built with a fundamental aspect in mind: modularity and extensibility.
You will find a lot of modules within this repository. All of them revolve around rre-core, which provides, as the name suggests, the core evaluation functionalities.
If, from one side, the runtime perspective of the rre-core is a library which can be used programmatically in your code, the ultimate goal is to provide several runtime containers for running the evaluation process within some managed environment (e.g. Apache Maven, Gradle, Apache Solr, Jenkins). One of these containers, the Apache Maven binding, is already available in the current release.
We usually don't hard-code things in RRE. The "open-close" approach is required in these kinds of tools because:
- it's not possible to create in advance a binding for all available search platforms
- likewise, we couldn't create an implementation for all available evaluation measures
With this in mind, we provided, where it makes sense, a level of abstraction which allows for binding new things in RRE.
RRE doesn't target a specific search platform. It provides, as mentioned before, an abstraction layer which defines, through interfaces, the expected lifecycle methods that a search platform must have in order to be integrated in a search quality evaluation process.
There are three types of lifecycles that are supported by RRE:
-
Full Lifecycle, meaning standing up the search engine from scratch with it's configuration, indexing the data into it, and then running the regression tests. Currently support is for Apache Solr and Elasticsearch (see the supported platforms section).
-
External Solr and Elasticsearch. In this situation we assume that the configuration and data has already been deployed, and we're just running the regression tests directly against these engines.
-
Custom Search Engine Adapter. You may have wrapped Solr and Elasticsearch in your own APIs, or even be using a complete different search engine altogether! We now provide an abstraction layer that lets you create your own Adapter.
Learn more in the Search Platform Framework section.
RRE is entirely written in Java, but it can be used in non-Java projects by means of the RRE Maven archetype or (not part of the current release) RRE CLI. That means your project can be written in any language, and in order to run RRE you must have
- a JVM (8 or later) and Apache Maven (for using the Maven archetype)
- a JVM (8 or later) for the command line client (not part of the current release)