Search Criteria

[demetriuscgu]

Hi,

I have been using IPED version 3 for some time. Now I decided to migrate to version 4 (4.0.1). However, I have noticed that some searches are not working: wildcard substitution, character substitution and RegEx. What am I doing wrong? The log file shows an exception caused by a missing library (jep), but it doesn't seem to be the cause of the problem. Anyway, I'll share the log file and all the search criteria used.

Thanks
Demétrius Borges.

IPED-SearchApp.log

[wladimirleite]

@lfcnassif can provide a better anwser, but it seems to be a bug.
I made a quick test here, and if uppercase letters are used together with wildcards, it doesn't find any hits.
For example:
car, Car, CAR, ca?, ca* work (last two can give different results, obviously).
Ca*, CA*, cA*, CA? are not working.

[demetriuscgu]

Thanks @tc-wleite. It really works when use lowercase.

[demetriuscgu]

Just adding another information: although the search worked with lowercase letters, it does not show the findings in the occurrences panel.

[lfcnassif]

Thanks @demetriuscgu for reporting and thanks @tc-wleite for showing exactly how to reproduce. This is a regression indeed, I'll take a look in a couple of hours.

About the second issue reported by @demetriuscgu, it works when selecting/switching to the Text viewer, seems a minor regression, should be easy to fix.

[paulobreim]

Testing here:
car, Car, CAR, ca?, ca* work (works well).
ca* returned 989.039
Ca* returned 35 but all occurrences is Ҁā just 1 time
CA* returned 2 occurrences with word camila, one of then is the file name
cA* returned 26 occurrences with differences words like Ć ć
CA? empty

[lfcnassif]

Taking a closer look, this was caused by commit e3edec2. When upgrading to Lucene-7.x on the road for 9.x, they removed the StandardQueryParser.setLowercaseExpandedTerms(boolean) method (see https://issues.apache.org/jira/browse/LUCENE-7355). Reading the migration guide (https://lucene.apache.org/core/7_0_0/MIGRATE.html) I understood it would work out of the box automatically, but seems the Analyzer default implementation should be overridden, this change should have been tested better...

[lfcnassif]

I opened #1246 and #1247 to track the progress on the issues. And I just pushed proposed fixes for them to #1246_#1247_fixes branch. Actually the first reported issue was easy to fix and the highlighting one was harder, I tried several approaches because Lucene Query.rewrite() method stopped to work as it used to...

For those affected, please test the snapshot that is available here:
https://github.com/sepinf-inc/IPED/actions/runs/2785468244

You don't need to reprocess the case, just replace the iped-engine-4.0.1.jar file inside the case/iped/lib folder.

Scoring for wildcard, prefix, regex and range queries is not good, I'll try to improve the computation a bit more tomorrow.

[lfcnassif]

I'll merge the proposed fixes and try to improve the scoring for MultiTermQuery queries in a separate ticket. If anyone could help testing the snapshot and give feedback, please let me know if the fixes are good or not.

[paulobreim]

Tested again and now it is ok.

[lfcnassif]

The snapshot or 4.0.2? Anyway, thanks for testing.

[paulobreim]

4.0.2 with ca* Ca*, etc..
if you need any test, I have a big machine (2 processors with 16 cores each, 384GB RAM) dedicated for test (and work), but 24 hours available.

[lfcnassif]

Thanks @paulobreim

[paulobreim]

Now I have also indexed with snapshot.
Need some special test?

[lfcnassif]

There is no need @paulobreim, the changes in snapshot above are already included in 4.0.2, thanks.

[demetriuscgu]

I did some tests with version 4.0.2 and everything worked perfectly: score, occurrences. Great job and thank you all for the assistance.

[lfcnassif]

Thanks for testing.