Sentence segmentation

[dsmiley]

It would be neat to add some sort of sentence segmentation to the query time text analysis to trigger a break in tagging. For example (a very silly one!) the input document text is:
" I want to buy something new. England is a nice place to visit. " Then assuming "New England" is in the dictionary (and possibly England but that doesn't matter) then the tagger will currently find "New England" which is undesirable. Of course this is a "naive tagger" as put to me when I joined the project; but nonetheless this sort of rule seems to me a good one to have at this layer in an overall system.

This could be implemented with a tokenizer that tokenizes sentences using Java's BreakIterator. It would set a new attribute that indicates the starting and ending offset of the sentence. Then the token would get split by other standard lucene components like WordDelimiterFilter which breaks on whitespace. Ultimately the Tagger could look for the custom attribute, and check if the last word's offsets don't fall within the current sentence as indicated by the attribute.

But maybe sentence segmentation isn't aggressive enough. After all, shouldn't there be a tag break at nearly any punctuation?

[mubaldino]

Could be language specific, genre specific, etc.

I don't think the tagger can deal with this, unless you are willing to take as an input the set of sentence spans. Otherwise, you being the tagger and one component of a larger pipeline would be dictating such segmentation. If you provide the option, you should expose it fully:

• allow caller to send in segment info
• allow caller to turn on SolrTextTagger segmenter, with option to report detected segments

otherwise it would seem half-baked to be making such decisions internal to tagger.

-m

[dsmiley]

I think that if some other component of a larger pipeline wanted to do segmentation, perhaps because it's more sophisticated, then the feature I propose here simply wouldn't be used. The BreakIterator based segmenter is better than nothing for those that don't have the time/expertise to develop the language specific, genre(?) specific segmenter.

What is your thoughts on breaking on all punctuation that's next to whitespace? It would have to be configurable of course, like perhaps not breaking on a hyphen.

[sujitpal]

Did you check that this actually happens? I just tried this sentence "All new. Norfolk gets a facelift" (because there is a New Norfolk (id:2155415) in the cities list) and it returns the following result (reformatted to return ("id", "name", "startOffset", "endOffset", "matchedSubstring")).

4776222 Norfolk 9 16 Norfolk
4945453 Norfolk 9 16 Norfolk
5073965 Norfolk 9 16 Norfolk
5128862 Norfolk 9 16 Norfolk

Assuming it does, we do something in user-land that might be of interest...

Sometimes we need to annotate at the sentence level, and doing a sentence per call works out to be too expensive (network overhead). So we create synthetic docs, say consider a max sentence size of maybe 500 chars and then join a bunch of sentences (say 50 or so) each space padded to 500 chars. Since the starting offset for each sentence is known, we can identify annotations based on the start and end offsets reported and also get the matching text from each sentence. We do this for performance, but one could use the technique to do sentence segmentation outside SolrTextTagger as well.

[dsmiley]

I'm sure "this can happen"; I could add a trivial test.

Nice trick on the sentence alignment. What could be useful is an additional TokenFilter that recognizes a large jump in startOffset, and if so increments the positionIncrement attribute, thus thwarting a match across the large span. If the Tagger sees a position increment > 1 (and if ignoreStopwords is false), it breaks any tags in progress (line ~128 of Tagger.java). ignoreStopwords has a dynamic default dependent on the presence of a StopWordFilter on the index analysis chain on the field.

[simonatdrg]

In trying to reduce network overhead for tagging multiple separate entities (separate text fields in our case) in a single call, we did something similar ( see #35 ) , inserting a junk token which would never be matched between the concatenated entities, then using the returned offsets to demultiplex the returned tagger matches.

[dsmiley]

Great suggestion @simonatdrg ; LOL its much simpler than my idea of the TokenFilter :-)

[sujitpal]

Yes, my field type is little different from the one suggested in the QUICK_START.md. That is why I wasn't sure if you saw the same thing I was seeing. As you pointed out, I guess the positionIncrementGap setting is working for me here.

In addition to the positionIncrementGap I also have omitTermFreqAndPositions set. I copied these off my previous configuration (versions Solr 5.0.0 and SolrTextTagger 2.3-SNAPSHOT). Not sure if these used to be the recommended settings earlier, although I do remember doing some tuning based on my use case at the time.

Also my analysis chain is a little different since I explicitly handle mixed case and lower case (in my application). Also my name_tag analog is multiValued since I explicitly populate it from code (i.e. not a copy-field directive) using all my names (primary + all alternates).

"add-field-type":{
"name" : "tag",
"class" : "solr.TextField",
"positionIncrementGap" : "100",
"postingsFormat" : "Memory",
"omitTermFreqAndPositions" : true,
"omitNorms":true,
"indexAnalyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.ASCIIFoldingFilterFactory"},
{"class":"solr.EnglishPossessiveFilterFactory"},
{"class":"org.opensextant.solrtexttagger.ConcatenateFilterFactory"}
]},
"queryAnalyzer":{
"tokenizer":{
"class":"solr.StandardTokenizerFactory" },
"filters":[
{"class":"solr.ASCIIFoldingFilterFactory"},
{"class":"solr.EnglishPossessiveFilterFactory"}
]}
},