Compiling 4.1.0 on Linux, failing testRecycleBinInfo2 and errors in testOCRParser

[reacan]

Hello everyone!

Congratulations on the 4.1.0 release to all the hard working developers!

I'm trying to build 4.1.0 on Linux and I am experiencing problems with tesseract and probably rifiuti2 packages.

Results :

Failed tests: testRecycleBinInfo2(iped.parsers.external.ExternalParserTest)

Tests in error:
testOCRParserTIFF(iped.parsers.ocr.OCRParserTest): tesseract returned error code 1
testOCRParserPDF(iped.parsers.ocr.OCRParserTest): tesseract returned error code 1
testOCRParserPNG(iped.parsers.ocr.OCRParserTest): tesseract returned error code 1
testOCRParserPSD(iped.parsers.ocr.OCRParserTest): tesseract returned error code 1
testOCRParserSVG(iped.parsers.ocr.OCRParserTest): tesseract returned error code 1

Tests run: 175, Failures: 1, Errors: 5, Skipped: 0

Full log: https://pastebin.com/QwcPyYys

I installed tesseract 5.0.0-alpha-20210401 but it did not help. When I tried compiling with tesseract 4.1.1-2.1 only two tests failed - PDF and PNG.
As for rifiuti2 I have version 0.7.0-3 installed.

Could this be caused by incompatible package versions, and if so what versions of the aforementioned packages should I try?

[lfcnassif]

Hi @reacan,

The testRecycleBinInfo2 failure I think is related with timezone differences. Could you run rifiuti2 manually on file below and post the results here?
https://github.com/sepinf-inc/IPED/blob/master/iped-parsers/iped-parsers-impl/src/test/resources/test-files/test_recyclebinINFO2

The OCR test errors I think are related to missing language packs. Try installing the english (eng) and portuguese (por) ones.

[reacan]

Hi @lfcnassif,

You were right about the OCR, installed portuguese and tesserect passed! Now only the "testRecycleBinInfo2" is failing.

This is the output I got from rifiuti2:

Recycle bin path: 'test_recyclebinINFO2'
Version: 5
OS Guess: Windows XP or 2003
Time zone: Coordinated Universal Time (UTC) [+0000]

Index Deleted Time Gone? Size Path
44 2008-10-28 15:53:42 No 4096 C:\Documents and Settings\All Users\Desktop\有道桌面词典.lnk
45 2008-11-03 15:01:59 No 4096 C:\Documents and Settings\Administrator\Desktop\wongsir_url.txt
46 2008-11-06 09:20:58 No 2912256 C:\Documents and Settings\Administrator\Desktop\dd-wrt.v24_mini_wrt54g.bin
47 2008-11-13 12:08:39 No 765952 C:\Documents and Settings\Administrator\Desktop\theme.svn
48 2008-11-13 12:11:33 No 5812224 C:\Documents and Settings\Administrator\Desktop\Config Client
49 2008-11-13 12:11:36 No 1847296 C:\Documents and Settings\Administrator\Desktop\Config Client.7z
50 2008-11-19 04:42:04 No 4096 C:\Documents and Settings\All Users\Desktop\Wireshark.lnk
57 2008-11-19 05:07:15 No 2727936 C:\Documents and Settings\Administrator\Desktop\GetDataBackforFAT-v3.63_PConline.rar
64 2008-11-19 05:07:35 Yes 2727936 C:\Documents and Settings\Administrator\Desktop\GetDataBackforFAT-v3.63_PConline
65 2008-11-19 05:17:12 No 4096 C:\Documents and Settings\Administrator\Desktop\360保险箱.lnk
66 2008-11-19 05:21:37 No 2732032 C:\Documents and Settings\Administrator\Desktop\gdb
67 2008-11-19 05:21:37 No 2723840 C:\Documents and Settings\Administrator\Desktop\gdb.zip
68 2008-11-19 11:34:23 No 0 C:\Documents and Settings\Administrator\Desktop\recovered files
69 2008-11-19 18:51:45 No 2727936 C:\Documents and Settings\Administrator\Desktop\GetDataBackforFAT-v3.63_PConline
70 2008-11-19 18:51:45 No 5169152 C:\Documents and Settings\Administrator\Desktop\Uneraser_Setup(2).exe
71 2008-11-19 18:51:45 No 5169152 C:\Documents and Settings\Administrator\Desktop\Uneraser_Setup.exe

[lfcnassif]

Do you have rifiuti (not rifiuti2) on your system?

[reacan]

Unfortunately it still fails:

testRecycleBinInfo2(iped.parsers.external.ExternalParserTest) Time elapsed: 0.012 sec <<< FAILURE!
java.lang.AssertionError
at org.junit.Assert.fail(Assert.java:87)
at org.junit.Assert.assertTrue(Assert.java:42)
at org.junit.Assert.assertTrue(Assert.java:53)
at iped.parsers.external.ExternalParserTest.testRecycleBinInfo2(ExternalParserTest.java:223)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

Results :

Failed tests: testRecycleBinInfo2(iped.parsers.external.ExternalParserTest)

[reacan]

Do you have rifiuti (not rifiuti2) on your system?

I have both. I will uninstall rifiuti and give it another go!

[reacan]

Uninstalled rifiuti and everything worked like a charm! Thank you very much for the help 😄

[lfcnassif]

You are welcome!

[reacan]

Btw this is possibly still not resolved. To succesfully compile 4.1.2 on Linux I had to make a symlink rifiuti2 > rifiuti. The test during compilation did not recognize rifiuti2 command.

[arisjr]

@reacan @lfcnassif On the iped-docker Dockerfile.dependencies we make a link for some executables, rifiuti is one of them.

update-alternatives --install /usr/bin/rifiuti rifiuti /usr/bin/rifiuti2 1

For more information on the dependencies build, please take a look on https://github.com/iped-docker/iped/blob/master/Dockerfile.dependencies

[lfcnassif]

we make a link for some executables, rifiuti is one of them.

But this used to be done before #1529 was closed, right?

@patrickdalla, since you are a Linux user, the build with tests are working fine without renaming rifiuti2 on your environment?

PS: The build on the remote CI (ubuntu 22.04) is working fine.

[lfcnassif]

@reacan what is the rifiuti2 version you are using? On Windows we use version 0.7.0

[lfcnassif]

@reacan, I took a closer look. Actually #1529 fix was applied just on master branch. I just copied it to 4.1.x branch, please try to build that branch.

[lfcnassif]

I remembered why the change wasn't included in 4.1.x at first. Since we renamed the windows exe to rifiuti2.exe, if some user has a custom profile with a custom conf/ExternalParsers.xml file, INFO2 parsing would stop working because it wouldn't find the old tools/rifiuti.exe. Possibly I will revert the changes on 4.1.x and they will be delayed to 4.2.0.

[reacan]

hi @lfcnassif, I have rifiuti2 0.7.0-3 installed.

I tried to build 4.1.x branch, and during the building there were no errors however when I launched processing there was an error message:

2023-06-05 23:50:44 [ERROR] [parsers.external.ExternalParsersConfigReader] Error testing RecycleInfo2Parser. Disable it in conf/ExternalParsers.xml or install 'rifiuti'

Creating a symlink rifiuti2 > rifiuti solves the error message.

Probably need to process an actual image to test if it works.

[lfcnassif]

2023-06-05 23:50:44 [ERROR] [parsers.external.ExternalParsersConfigReader] Error testing RecycleInfo2Parser. Disable it in conf/ExternalParsers.xml or install 'rifiuti'

If that message didn't show up when building, maybe an old config file was used when processing. Sorry for asking again, but have you cleaned up the previous build before building again?

[lfcnassif]

Checking the command present in your conf/ExternalParsers.xml is a way of verifying that.

[reacan]

My bad, I was using the old ExternalParses.xml not the updated file. Everything works smoothly now, thank you!

[lfcnassif]

Great! Unfortunately this exact same problem can happen to Windows users with custom profiles, so I'll have to revert the change:

I remembered why the change wasn't included in 4.1.x at first. Since we renamed the windows exe to rifiuti2.exe, if some user has a custom profile with a custom conf/ExternalParsers.xml file, INFO2 parsing would stop working because it wouldn't find the old tools/rifiuti.exe. Possibly I will revert the changes on 4.1.x and they will be delayed to 4.2.0.