Apply highlight and matches only on attributes that are searchable

[curquiza]

Following meilisearch/meilisearch#1864

Currently, even if attributes are not set in searchableAttributes, the highlight and matches still apply on them.

Example using MeiliSearch v0.23.1

• Add this document

{
  "id": 1,
  "title_fr": "Le Petit Prince",
  "title_en": "The Little Prince"
}

• Set only title_fr as searchableAttributes

• Do the following search

{
    "q": "prince",
    "attributesToHighlight": ["*"],
    "matches": true
}

Here is the following response:

{
    "hits": [
        {
            "title_fr": "Le Petit Prince",
            "id": 1,
            "title_en": "The Little Prince",
            "_formatted": {
                "title_fr": "Le Petit <em>Prince</em>",
                "id": "1",
                "title_en": "The Little <em>Prince</em>"
            },
            "_matchesInfo": {
                "title_en": [
                    {
                        "start": 11,
                        "length": 6
                    }
                ],
                "title_fr": [
                    {
                        "start": 9,
                        "length": 6
                    }
                ]
            }
        }
    ],
    "nbHits": 2,
    "exhaustiveNbHits": false,
    "query": "prince",
    "limit": 20,
    "offset": 0,
    "processingTimeMs": 0
}

Despite only title_fr is considered as searchable, title_en is also highlighted and
"matched".

We might consider only applying highlight/matches to the searchable attributes.
Currently, if the document is retrieved, all the query words are highlighted. But they don't represent the words that really made the document match.
We might need more user feedback to understand what is the most expected behavior.

[gmourier]

For now, searchableAttributes is completely uncorrelated with attributesToCrop or attributesToHighlight, which makes sense at first. _formatted is there to provide visual help within the documents retrieved. This can be used to help end-users to find documents in the results that contain part of the query terms if highlighting is requested on those fields.

Today, it is possible to apply highlighting only on searchableAttributes. To do so, searchableAttributes just have to be set as attributesToHighlight during a search.

However _matchesInfo is not linked to attributesToHighlight or attributesToCrop. Its current behavior is to display for any field matching some of the query terms, the positions where a field matches those terms. It could be linked to only display matches for field set in attributesToHighlight or attributesToCrop.

This needs more thought. I have the impression that matches are mostly used to override <em> tags when applying highlight as they are not customizable.

In the idea of understanding users better, why do you use _matchesInfo today? 🤓

[Thearas]

This needs more thought. I have the impression that matches are mostly used to override tags when applying highlight as they are not customizable.

That's why I don't use attributesToHighlight. Our documents may be in Markdown format. 😂

[CaroFG]

I came across this issue today, I was tempted to suggest using _matchesInfo to crop his text, but there's no way to know where the exact match of the phrase search start and ends.
There could be an additional field in _matchesInfo for phrase search

[gmourier]

Thanks, @CaroFG, that's an interesting idea! MeiliSearch could also separate the terms to be highlighted within a search phrase that matches in a document in comparison to "simple" terms and do the same in _matchesInfo.

Currently, this is what happens if q = "The seeker seeks the one".

{
    "hits": [
        {
            "_formatted": {
                "searchable_field": "This is a long text. <em>The</em> <em>seeker</em> <em>seeks</em> <em>the</em <em>one</em> exact phrase search. <em>The</em> car is red."
            },
            "_matchesInfo": {
                "searchable_field": [
                    {
                        "start": 22,
                        "length": 6
                    },
                    {
                        "start": 43,
                        "length": 2
                    },
                    {
                        "start": 46,
                        "length": 3
                    },
                    {
                        "start": 50,
                        "length": 3
                    },
                    {
                        "start": 54,
                        "length": 5
                    },
                    {
                        "start": 101,
                        "length": 3
                    }
                ]
            }
        }
    ],

}

Note, that _matchesInfo is following the same rule.

It could be something like that when we encounter an exact phrase search within the q parameter.

{
    "hits": [
        {
            "_formatted": {
                "searchable_field": "This is a long text. <em>The seeker seeks the one</em> exact phrase search. The car is red."
            },
            "_matchesInfo": {
                "searchable_field": [
                    {
                        "start": 22,
                        "length": 24
                    }
                ]
            }
        }
    ],

}

While searching for an exact phrase search and a term. E.g. "The seeker seeks the one" the could be

{
    "hits": [
        {
            "_formatted": {
                "searchable_field": "This is a long text. <em>The seeker seeks the one</em> exact phrase search. <em>The</em> car is red."
            },
            "_matchesInfo": {
                "searchable_field": [
                    {
                        "start": 22,
                        "length": 24
                    },
                    ...  //_matchesInfo for `The` query term
                ]
            }
        }
    ],

}

This way, we do not have to add an additional field for an exact phrase search. What do you think of this idea?

[CaroFG]

So if I understand it correctly:
If q = "The seeker seeks the one"
MeiliSearch would only highlight and return matches info for the search terms enclosed in double-quotes, which means, the words in the order given by the search query. So the last "the" in:

"This is a long text. The seeker seeks the one exact phrase search. The car is red."

wouldn't be present in the formatted object nor in the matchesInfo object. This seems logical to me.

If q = "The seeker seeks the one" the

This time the last "the" would be present in the formatted object and in the matchesInfo object.

But what happens if the document content is the following?

"The text is long. The seeker seeks the one exact phrase search. The car is red."

Here we've got 2 "the" one before and one after the phrase search. How do we know which are the results for the phrase search in the matchesInfo? Are the phase search results always going to appear first?

[gmourier]

In this case, we could have _formatted returning highlight as it <em>The</em> text is long. <em>The seeker seeks the one</em> exact phrase search. <em>The</em> car is red with 3 objects in the _matchesInfo` like that:

"_matchesInfo": [
     { // first `the` },
     { //phrase search },
     { //second the at the end}
]

The engine may be able to have a rendering of highlight/matches depending on the type of query objects(term or phrase search) matching. I'm not sure if it's useful to indicate in _matchesInfo object that it concerns a phrase search as long as the matches don't overlap between a phrase search and a query term.

Is it really important to know which _matchesInfo belong to a phrase search? I may be missing something 😇

[CaroFG]

Yes, IDK, maybe they want to crop their document according to the position of the phrase search. But maybe the first solution you presented, solves the problem...