XZ lib compatiblity

[Nephos]

Hello,

I'm trying to make xz work but there is a problem I'm trying to debug.
Any idea already about how to fix it?

Linux xxx 6.1.35-1-lts #1 SMP PREEMPT_DYNAMIC Wed, 21 Jun 2023 15:54:02 +0000 x86_64 GNU/Linux
truffleruby 22.3.1, like ruby 3.0.3, GraalVM CE Native [x86_64-linux]
truffleruby --debug -e "require 'xz'" 

Exception: `LoadError' at <internal:core> core/kernel.rb:236:in `gem_original_require' - cannot load such file -- xz
Exception: `RuntimeError' at /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/cparser.rb:211:in `parse_ctype' - unsupported type: uint32_t
/home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/cparser.rb:211:in `parse_ctype': unsupported type: uint32_t (RuntimeError)
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/cparser.rb:117:in `block in parse_signature'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/cparser.rb:117:in `collect'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/cparser.rb:117:in `parse_signature'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/fiddle/import.rb:168:in `extern'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/gems/gems/ruby-xz-1.0.0/lib/xz/lib_lzma.rb:154:in `<module:LibLZMA>'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/gems/gems/ruby-xz-1.0.0/lib/xz/lib_lzma.rb:34:in `<module:XZ>'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/gems/gems/ruby-xz-1.0.0/lib/xz/lib_lzma.rb:28:in `<top (required)>'
        from <internal:core> core/kernel.rb:293:in `require_relative'
        from /home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/gems/gems/ruby-xz-1.0.0/lib/xz.rb:526:in `<top (required)>'
        from <internal:core> core/kernel.rb:234:in `gem_original_require'
        from <internal:/home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
        from -e:1:in `<main>'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- xz (LoadError)
        from <internal:/home/xxx/.rbenv/versions/truffleruby-22.3.1/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:core> core/unbound_method.rb:18:in `bind_call'
        from <internal:core> core/kernel.rb:272:in `require'
        from -e:1:in `<main>'

regards!

Edit: saw the fiddle incompatibility, I suppose it will be hard to replace or patch

[eregon]

Thanks for the report.

Edit: saw the fiddle incompatibility, I suppose it will be hard to replace or patch

What do you mean by fiddle incompatibility?

Fiddle should work, but there is probably a missing type or so in our fiddle backend.

[Nephos]

What do you mean by fiddle incompatibility?

I was thinking about #1572

I'm looking at this missing type right now, still learning how this is working as a whole for now. If you have any lead I probably can propose a patch

I guess it's somewhere around https://github.com/oracle/truffleruby/blob/master/lib/truffle/truffle/ffi_backend/type.rb#L257

[eregon]

With truffleruby-dev I get (xz is https://github.com/win93/ruby-xz):

$ gem i ruby-xz
$ ruby -v -rxz -e 'p XZ'
truffleruby 23.1.0-dev-f3cb0e84, like ruby 3.1.3, GraalVM CE Native [x86_64-linux]
/home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/fiddle_backend.rb:63:in `type_to_nfi': -6 not implemented (NotImplementedError)
	from /home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/fiddle_backend.rb:219:in `block in initialize'
	from /home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/fiddle_backend.rb:219:in `map'
	from /home/eregon/.rubies/truffleruby-dev/lib/truffle/truffle/fiddle_backend.rb:219:in `initialize'
	from /home/eregon/.rubies/truffleruby-dev/lib/mri/fiddle/import.rb:299:in `import_function'
	from /home/eregon/.rubies/truffleruby-dev/lib/mri/fiddle/import.rb:170:in `extern'
	from /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/ruby-xz-1.0.3/lib/xz/lib_lzma.rb:156:in `<module:LibLZMA>'
	from /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/ruby-xz-1.0.3/lib/xz/lib_lzma.rb:34:in `<module:XZ>'
	from /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/ruby-xz-1.0.3/lib/xz/lib_lzma.rb:28:in `<top (required)>'
	from <internal:core> core/kernel.rb:293:in `require_relative'
	from /home/eregon/.rubies/truffleruby-dev/lib/gems/gems/ruby-xz-1.0.3/lib/xz.rb:533:in `<top (required)>'
	from <internal:core> core/kernel.rb:234:in `gem_original_require'
	from <internal:/home/eregon/.rubies/truffleruby-dev/lib/mri/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:core> core/unbound_method.rb:18:in `bind_call'
	from <internal:core> core/kernel.rb:272:in `require'
<internal:core> core/kernel.rb:236:in `gem_original_require': cannot load such file -- xz (LoadError)
	from <internal:/home/eregon/.rubies/truffleruby-dev/lib/mri/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:core> core/unbound_method.rb:18:in `bind_call'
	from <internal:core> core/kernel.rb:272:in `require'

[eregon]

Also if for some reason the fiddle gem is installed then it causes this error instead:

$ ruby -v -rxz -e 'p XZ'
truffleruby 23.1.0-dev-f3cb0e84, like ruby 3.1.3, GraalVM CE Native [x86_64-linux]
/home/eregon/.rubies/truffleruby-dev/lib/cext/include/ruby/internal/arithmetic/long.h:124:in `Init_fiddle': Attempt to wrap something that isn't an Ruby object (ArgumentError)

We should change fiddle gem upstream so it does nothing on TruffleRuby, and add TruffleRuby in CI if possible, ref #2644.

[Nephos]

Just for information (I don't understand yet what all of this imply), with the standalone I got

bin/truffleruby -v -rxz -e 'p XZ'
truffleruby 23.1.0-dev-f3cb0e84, like ruby 3.1.3, Interpreted JVM [x86_64-linux]
/home/xxx/graal/sdk/mxbuild/linux-amd64/GRAALVM_1E2AC487B0_JAVA17/graalvm-1e2ac487b0-java17-23.1.0-dev/languages/ruby/lib/truffle/truffle/fiddle_backend.rb:63:in `type_to_nfi': -6 not implemented (NotImplementedError)

[eregon]

I have a fix in #3129, it's quite trivial, we just didn't handle those types yet in the case/when.
There are tests for parsing long long in a signature but not for calling a function using such a type (maybe there is no libc function using long long or uint64_t?).

[Nephos]

I don't remember everything about long long but I think it's something nobody on x86_64 ever use since "long" is sufficient
Agree I was about to send you the same patch

Edit: gnu libc says long long is the way to define 64 bit numbers

[eregon]

I don't remember everything about long long but I think it's something nobody on x86_64 ever use since "long" is sufficient

Right, it's because Fiddle parses uint64_t in the same case as long long, so they are treated the same:

truffleruby/lib/mri/fiddle/cparser.rb

Line 163 in 38b5987

when /\A(?:unsigned\s+long\s+long(?:\s+int\s+)?|uint64_t)(?:\s+\w+)?\z/

[Nephos]

I should not have closed this quickly, the original problem was on uint32_t not 64. I'll keep investigating a bit on that.

[eregon]

Note 22.3.1 is a bit old by now, please always use the latest release (23.0.0 currently).