Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build a library usable for embedding & FFI #492

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

byteit101
Copy link

Using a library via FFI is usually done via a shared library load. When I was attempting to embed HashLink via an FFI, I had a need for a shared library version of hl so that I could use the functions for loading and initializing .hl bytecode. This PR adds make targets for such a library, plus exposing a few more methods necessary when looking up and calling objects as I've seen referenced in the community and issues here. Additionally, there is a thread-helper method I found necessary when calling from not just a single spot in the C stack. Currently, I make it build as libhl-jit.so, but I'm not married to the name. I originally was thinking libhl-embed.so.

Things not in this PR, but could be helpful for embedding via an FFI:

It may be helpful to also document how to integrate with the GC, that took me a while to figure out.

An entrypoint helper. It's a bit more verbose in non-C languages via FFI

Method lookup/calling helpers. This surprised me at the lack of helpers. Right now my FFI code looks like this for driving looking up methods:

def lookup_function(name, type, instance)
	hash = field_hash(name.to_s)
	lookup = case type.kind
		when Hl::HOBJ
			Hl.obj_resolve_field(type.details, hash)
		when Hl::HVIRTUAL
			tv = Hl::TypeVirtual.new(type.details) # type cast, not instantiation
			Hl.lookup_find(tv.lookup, tv.nfields, hash)
		else
			raise "TODO: implement lookup for #{type.kind}"
	end
	raise NameError.new("Method doesn't exist #{name}") if lookup.null?
	fl = Hl::FieldLookup.new(lookup) # type cast, not instantiation
	return Hl.dyn_getp(instance, fl.hashed_name, Hl.dyn)
end

I didn't implement these as I wanted to discuss them first, after a minimum basic solution (the PR right now).

This enables library-based consumers (ffi, etc) to easily
embed hashlink without recompiling the code
@ncannasse
Copy link
Member

hi !

I think you need a bit more than hl_enter_thread_stack in order to correctly init the VM.
You need to call both hl_global_init() and hl_register_thread()

As for the helpers, you're right the VM was designed to be used by compiler generated code, not directly with FFI, so the helpers we have are mostly focused on wrapping C primitives for VM, not for the user to manage the VM objects directly.

@byteit101
Copy link
Author

Yes, sorry I didn't provide enough context. I was assuming the other bits were already called. Here is my current usage of enter_thread_stack and the global init. Note the enter_thread_stack is called per-call (call_raw), while the read init/register is just once

def initialize(bytecode)
	Hl.global_init
	Hl.sys_init(nil, 0, nil) # TODO: args?  #hl_sys_init((void**)argv,argc,file);
	# the GC stops at the stack pointer. Don't scan any ruby code
	Hl.register_thread(nil)
	
	...
end

def dispose
	Hl.module_free(@m)
	alloc_field = FFI::Pointer.new(:pointer, @code.pointer.address + @code.offset_of(:alloc))
	Hl.free alloc_field #(@code.alloc)
	Hl.unregister_thread
	Hl.global_free
end

def call_raw(closure, args)
	Hl.profile_setup(-1) # TODO: profile setup?
	# avoid the ruby&ffi stack, we must keep this updated for each call
	Hl::enter_thread_stack(0)

	ret = if args == []
		Hl.dyn_call_safe(closure,nil,0,@isExc)
	else
		# TODO: HL_MAX_ARGS!
		HashLink.memory_pointer(:pointer, 10) do |cargs|
			cargs.write_array_of_pointer(args)
			Hl.dyn_call_safe(closure,cargs,args.length,@isExc)
		end
	end
	Hl.profile_end
	if @isExc.get_int8(0) != 0
		a = Hl.exception_stack
		hlstr = Hl.to_string(ret).read_wstring
		ex = ::HashLink::Exception.new("Uncaught HashLink exception: #{hlstr}")
		ex.set_backtrace(Hl::Varray.new(a).asize.times.map do |i|
			_read_array(a, i).read_wstring
		end)
		raise ex
	end
	return ret
end			

I'm still working on cleaning it up, but will be publishing the full FFI source shortly

@skial skial mentioned this pull request Nov 17, 2021
1 task
@byteit101
Copy link
Author

Here is how I'm using this, with full versions of the methods pasted above as examples: https://github.com/byteit101/hashlink-embed/blob/8288463a76bd520920f10b1eec69f98ca496e2aa/lib/hashlink-embed/hashlink.rb#L118

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants