Calling *CompiledFunction from Go

[tgascoigne]

I'm trying to load a script and then call a tengo function by name. I'm currently doing something like this:

script := tengo.NewScript(source)
compiled, err := script.RunContext(ctx)
fn := compiled.Get("someFunction")
fn.Call( ... )

It looks like fn is a *CompiledFunction. Its CanCall method returns true, but it doesn't appear to actually implement the Call function, only via its embedding of ObjectImpl.

Is it possible to call into a *CompiledFunction from Go?

[ozanh]

Hi, it is not possible to call *CompiledFunction from Go for v2. CanCall() and Call() methods are used in VM and let you provide callable to scripts from any type implementing them. See this page
Let me know your use case so I may suggest some workarounds.

[tgascoigne]

Hey, thanks for the reply. What I'm trying to do is essentially use Tengo functions as callbacks. So the user provides a script with a few named functions, which the Go code will call into when certain things happen.

[ozanh]

Here is a simple snippet for you I hope it helps;

package main

import (
	"log"

	"github.com/d5/tengo/v2"
	"github.com/d5/tengo/v2/stdlib"
)

func main() {
	src := `
text := import("text")

m := {
	contains: func(args) {
		return text.contains(args.str, args.substr)
	}
}
out := undefined
if f := m[argsMap.function]; !is_undefined(f) {
	out = f(argsMap)
} else {
	out = error("unknown function")
}
`
	script := tengo.NewScript([]byte(src))
	script.SetImports(stdlib.GetModuleMap("text"))
	argsMap := map[string]interface{}{
		"function": "contains", "str": "foo bar", "substr": "bar"}
	if err := script.Add("argsMap", argsMap); err != nil {
		log.Fatal(err)
	}
	c, err := script.Run()
	if err != nil {
		log.Fatal(err)
	}
	out := c.Get("out")
	if err := out.Error(); err != nil {
		log.Fatal(err)
	}
	// If it returns a dynamic type use type switch using out.Value()
	log.Println("result =", out.Bool())
}

OR you can create a script for each function.

[tgascoigne]

Thanks Ozan, that looks like a decent solution. My only concern is that I'm planning to have lots of these scripts, and this extra boilerplate around each of them would harm readability.

I've hacked together a solution which pokes the correct sequence of instructions into a *VM and pulls the return value back out at the end. It seems to work for some trivial test cases, and I'm planning to try it out in practice for a little while to solve my current problem.

I'd be fine with keeping this code on a private branch for my purposes (assuming it works in practice), but I wonder if you have any thoughts on this approach and if something similar may be accepted in a PR?

[ozanh]

Hi Tom, thank you for sharing your work. It will be hard for you to keep sync with upstream repository, I guess. I studied Bytecode today and created an example, which has some code from your hack and my Go love :). Code is not tested!
Here is the gist link to call *CompiledFunction from Go easily. As it is an example, no module support is added.

I used main script to define functions not source module. Return value of called function is set to a global variable instead of accessing VM.stack.
What do you think?
This is preview;

const mathFunctions = `
mul := func(a, b) {
	return a * b
}
square := func(a) {
	return mul(a, a)
}
add := func(a, b) {
	return a + b
}
`
func main() {
	s := NewScript()
	if err := s.Compile([]byte(mathFunctions)); err != nil {
		panic(err)
	}
	v, err := s.Call("add", 1, 2)
	if err != nil {
		panic(err)
	}
	fmt.Println(v)
	v, err = s.Call("square", 11)
	if err != nil {
		panic(err)
	}
	fmt.Println(v)
	v, err = s.Call("mul", 6, 6)
	if err != nil {
		panic(err)
	}
	fmt.Println(v)
}

[tgascoigne]

Nice! That looks perfect. I'll pull this into my local branch and have a play around with it tomorrow, thank you.

Perhaps this could be extended to support functions passed into Go from Tengo as well? Something like this:

m := import("some_builtin")

m.register(func() {
    // do some work
})

And then have the Go side call that func at a later time? I believe the first operand to OpCall can be any object which is callable, perhaps in addition to call by name we could expose another function with the signature Call(fn Object, args ...interface{})? Unsure if this would work correctly with closures though.

[ozanh]

I just revised the gist (rev3), I forgot to use only new instructions instead of adding to original bytecode.
I will check Call(fn Object, args ...interface{}) later but as you said, probably closures do not work. Currently, only CompiledFunction run.
Please see my old example

[tgascoigne]

Thanks again Ozan, I've integrated that with my code and it seems to work perfectly.

I've extended it a bit to support modules, and to pull in the Set, Get, SetImports etc. functions from tengo's Script type. I've also added a Callback type which wraps up a callable tengo.Object and a Script into one struct with a Call method, as I wanted to pass handles to these tengo functions around.

It seems to work fine with closures too, at least in my trivial test case.

Here's my final code: https://gist.github.com/tgascoigne/f8d6c6538a5841bcb5f135668279b93b

Many thanks for all your help!

[ozanh]

Thanks Tom, it is excellent. Everything works as expected with all features, why not. We only run new instructions with current constants and globals which looks harmless. Tengo functions can easily be called. It would be better if it did not run script once to get compiled functions from globals but anyway it works, may be later we can figure out how to get them easily from constants. If you allow me, I will create a new open source repo for this module and try to improve it in time using the final draft.

[tgascoigne]

It would be better if it did not run script once to get compiled functions from globals but anyway it works, may be later we can figure out how to get them easily from constants.

I think compiling and running the script ahead of time is the right thing to do - without it, I think things like non-constant globals would not be initialized correctly when referred to by functions. In my case, the overhead from doing so is not an issue as I'm storing the compiled result and reusing it multiple times.

If you allow me, I will create a new open source repo for this module and try to improve it in time using the final draft.

Please do :)

[ozanh]

I think compiling and running the script ahead of time is the right thing to do - without it, I think things like non-constant globals would not be initialized correctly when referred to by functions. In my case, the overhead from doing so is not an issue as I'm storing the compiled result and reusing it multiple times.

Yes you are right keeping as it is is only solution. After adding context support, interface{} conversion wrappers and documentation, couple of days later it will be released :). Finally, I can call tengo functions from my rule engine!

[d5]

Sounds like there were some meaningful outcomes here! Sorry that I missed the discussions, but please let me know if there's anything else I can do.

[ozanh]

@d5 I created a repo for calling Tengo functions from Go with @tgascoigne . Code is mostly ported from Tengo script.go file. Please have a look at https://github.com/ozanh/tengox. We need your expertise. Docs and examples are still missing, but tests look good.

[d5]

tengox is brilliant! 😄 Good job!

One thing I'd suggest is that you make them more portable or detached. A similar example would be the relationship between tengo.Script and tengo.Compiled. tengo.Script creates a new instance tengo.Compiled instance for each compilation. Then each tengo.Compiled instance can be used independently without affecting its parent tengo.Script.

Another thing we can consider is to bring that concept back to tengo.Script. Maybe we can add Script.CompileFunc(funcName) that returns CompiledFunc, which can be used in a similar way that Callback in tengox is used.

Just a thought.

[tgascoigne]

I'd love to see this included in Tengo, as there's a good chunk of code that had to be duplicated to make this work. If we were to change the API, there's a few qualities of Ozan's implementation I'd like to preserve:

• The ability to have a single handle which contains all of the information needed to invoke a function (*Callback). This is useful for storing functions and passing them around.
• The option to work with both named functions and tengo.Objects. This means we're not just limited to calling top level function definitions, but function literals and closures too.
• As we're currently executing the whole script as a prerequisite, we're able to make use of non-constant globals which get initialized along the way.

[ozanh]

@tgascoigne we need some time to integrate it in tengo, library must be mature enough to propose any changes. Although we can easily integrate this implementation to tengo passing callbacks to interfere compilation and run processes to change current behavior, we need some time.

We can change the tengox Script type to make it similar to tengo's. Separation as Script and Compiled enables to get rid of source code []byte by garbage collection if not required any more by user and makes maintenance easier. I can change signature of Script.CompileRun() error to Script.CompileRun() (*Compiled, error) this is trivial.

*Callback usage may bite some users by letting them to run them in Go side before VM finished and it can cause lock and lock the VM loop. For example;

        scr.Add("pass", &tengo.UserFunction{
		Value: func(args ...tengo.Object) (tengo.Object, error) {
			_, _, = scr.MakeCallback(args[0]).Call()
			return tengo.UndefinedValue, nil
		},
	})

Above usage looks idiomatic but pauses VM indefinitely due to mutex. We can return a promise object or a channel I don't know may be I am missing something but Murphy's law, it can happen. Please guide me.

Edit

I created a new branch, please check this out

[tdewolff]

Any advance on this issue? Would love to be able to execute Tengo functions from Go.

[misiek08]

PoC (I don't know any bugs, down-sides - didn't test in production, event not on sandbox) here: #372

I made this, because this repo looks more active than https://github.com/ozanh/ugo and those 2 projects are best in class for Go scripting which I need for new clients.

[weakpixel]

I'm playing around to implement games with go and I really feel in love with allowing to attach tengo scripts to the game objects. I also have a event system, which I want to be able to be used from within a tengo script.

I was think something like this:

init := func(node) {
  node.bind("myevent", func(e) {
        fmt.println(e)
  })
  node.x = ...
}

update := func(node) {
    ....
}

draw := func(node) {
    node.image.fillRect(...)
}

I realize this ticket is quite old, but was anything done to the tengo core to make it possible to work with callbacks?
Or is the example here (https://github.com/ozanh/tengox) still the way to go?

great work!