Multithreading

[TylerReynolds]

(Incidentally, i am in fact multi-threading, as this is my second thread, sorry xD)

Hey, sorry for being here asking something once again, but since you programmed the plugin it should take you 0.9 secs to help me with something i would struggle with for quite a few hours xD

Sssoo... I'm currently trying to keep a python script open while playing, because it loads some vectors in the RAM and it takes a couple minutes to initialize.
I really can't wait 5 minutes every time i call the script before being able to actually use it xD

The issue here is that whenever i load the python script, of course the game freezes, and even when the script is done loading the game cannot resume because i need to keep it open, and apparently script open = frozen.

What is your suggestion, what options do i have here?

Does this mean that i cannot have multiple actors executing python scripts, or that python scripts that require some time to compute would freeze the game every time?

The best option would be to be able to keep it open in unreal while playing, so that i can load it once when first starting the game, then i can send the script all the data it needs to elaborate without closing it.

It really just has to stay open and idle, no activity whatsoever, until i send it the data to elaborate on.
So basically i think i would just need a way to unfreeze the game once it is done loading, without stopping the script but keeping it idle and ready to go when needed.

So i think there really is no other way other than multithreading.

Grazie tante ancora una volta :D

[rdeioris]

Hi, you can enable multithreading (albeit pretty experimental as we are basically mixing 3 threading models) https://github.com/20tab/UnrealEnginePython#threading-experimental

or you can follow a 'coooperative' approach, where your python script periodically 'tick' the editor via unreal_engine.editor_tick() or uobject.world_tick() when you are in the game.

[TylerReynolds]

I also read something about subinterpreters, and the thread about the guy trying to create an instanced VM in an object so that he could create his hardware engineering game using multiple components to engineer something with, dunno if you remember.

Could subinterpreters be another option?

And by the way, how would i access the output from the python script, once it analyzes the data?
What would normally be the stdout.

I did read about the experimental multithreading, but the reason i asked was precisely because i didn't know just how experimental it is xD
Do i risk any issue in my project?
Is it safe if i use it just to "park" the script there, waiting for data to evaluate?

Additionally, i was very interested in a reply to this question if possible:
Does this mean that i cannot have multiple actors executing python scripts, or that python scripts that require some time to compute would freeze the game every time?
(And how would i solve such an issue?)

And, finally, could you elaborate on the second "cooperative" option you mentioned? I'm not quite sure i understand what exactly it means, how would i set it up, what could be some rough instructions to do it?
Because it does look like something that could be useful to check for new data, but i don't understand how it could be useful for keeping the python script open without having to stop it (to avoid the freezing of the game it causes).
The problem is that i need the python script open, because when it stops i have to reload the vectors into the RAM once again and that takes about 5 minutes :(
Do you mean that by calling uobject.world_tick() the game would resume, because i'm manually instructing it to do so?

Sorry for all the questions xD

[rdeioris]

There are lot of users that work with multithreading enabled in the plugin, so maybe it could work for you out of the box. Regarding 'blocking tasks' it is a limit in basically any game engine. You should offload every heavy task or your game will block. Threads are a way to do it, in other cases multiprocessing is another solution (at least on unix system where the copy on write fork() semantics allow for pretty funny tricks). Multiple interpreters are pretty useless in this concept.

The best thing to do is designing your script to work as 'tick' objects. So an actor (or a component) periodically update its status. To be a bit cleaner this would be a heavy approach:

class HeavyActor:

    def tick(self):
        for i in range(0, 10000):
            do_something_heavy()

that could be refactored to be game-engine friendly:

class HeavyActor:

    def begin_play(self):
        self.i = 0

    def tick(self):
        self.i += 1
        if self.i < 10000:
            do_something_heavy()

If for some reason you cannot work in this way the cooperative approach works this way:

import unreal_engine as ue

class HeavyActor:

    def tick(self):
        for i in range(0, 10000):
            do_something_heavy()
            self.uobject.get_world().world_tick()

[TylerReynolds]

Oh yes now i understand what you mean by tick objects, so basically check every x amount of time instead of every millisec because of course that would be unnecessary and too heavy.
So this part is clear.

About the last code example instead, does "self.uobject.get_world().world_tick()" make the world tick manually?
Like moving the hands of a clock by, well, hand xD to make time flow.
Is this what the method is used for?
Does time tick for the entire world or just the object i refer to, is there a way to tick it for the entire world?
I have some ideas about frozen time concepts if so, and this will definitely come in handy.

So to sum it up, my two options are to either use multithreading or to try manually ticking the time in my script so that it doesn't stay frozen, correct?
Are you recommending manually ticking the time as a better option than enabling multithreading?
(Also, how can i access the output from the script, which would normally be the sdtout?)

And how would i handle the multithreading once i enable it, is it automatically handled?
How would i setup the script to work in a secondary thread?

[rdeioris]

Yes, "ticking" the world manually moves it forward of the specified time slice (_tick() takes a float argument). I do not have a strong opinion between threading or 'playing with time', threading would be (maybe) a bit easier to integrate in already existing scripts.

Regarding the output i am not sure to follow, the stdout is mapped to the plugin console, but your scripts should return objects (like strings). If they are already developed scripts that blindly writes to stdout, you can use the FPythonOutputDevice class:

from unreal_engine import FPythonOutputDevice

def logger(message, verbosity, category):
    do_something_with_message()

output_device = FPythonOutputDevice(logger)

The 'logger' function will be called whenever a new message is sent to the console

[TylerReynolds]

So threading is handled automatically, right?
I just have to enable it and the script won't freeze the game anymore when called?

Oh, so scripts return objects.
But how can i access these objects, in a practical way?
My script executes a number of functions based on the data, then how could i access the results of the functions so that i can use them with unreal engine blueprints or in other scripts, for example?
If i were to not use a parsing of the sdtout but instead directly passing data around between scripts and blueprints (this would be the best way, so that i can act based on the data directly within unreal)

I also noticed sys.exit() doesn't work to end a script.
How can i do so?

[getnamo]

@TylerReynolds I've solved my multithreading use cases by adding two convenience functions to this plugin in a fork.

ut.run_on_bt(<function name>,<callback>) #should have a third parameter for args...

ue.run_on_gt(<function name>,<args>)

This allows me to run a long training function on a background thread, e.g.:

https://github.com/getnamo/tensorflow-ue4/blob/master/Content/Scripts/TensorFlowComponent.py#L92

and then give a game-thread callback when it has completed or weave multiple status updates while it's running without worrying about being on the wrong thread in UE4.

https://github.com/getnamo/tensorflow-ue4/blob/master/Content/Scripts/TensorFlowComponent.py#L83

I usually then handle data conversion using JSON encoding since SIOJson supports easy conversion between BP types and structs<->JSON.

The convenience functions are defined in: https://github.com/getnamo/UnrealEnginePython/blob/master/Source/UnrealEnginePython/Private/UEPyModule.cpp#L273

which calls the modified function:
https://github.com/getnamo/UnrealEnginePython/blob/master/Source/UnrealEnginePython/Private/UEPyEngine.cpp#L468

and background thread action using python Thread:
https://github.com/getnamo/UnrealEnginePython/blob/master/Content/Scripts/upythread.py#L19

(just realized it doesn't support calling bt with args in my use case, but that's an easy fix)

If these would be useful I could open a pull request with the two functions on the master repo.

[rdeioris]

@TylerReynolds i fear you are seeing it from an 'unclassical' point of view. You do not run python scripts like in a shell. The whole UE4 engine is a python vm that can execute code in a big context. You create classes that you attach to Actors, or you execute scripts in the UE4 python vm. Calling sys.exit() would mean destroying the editor itself, so it instead generates an exception so the result would be the ending of the script execution. Passing data between scripts is not an issue as if a script create the 'foobar' object, the other scripts will be able to see it.

If it is not the intended behaviout you have the sandbox_exec() (or 'Execute in Sandbox' button in the editor) that generate a new python interpreter that is destroyed at the end of the script execution

@getnamo i am not sure to understand why you did not use the create_and_dispatch_when_ready() function, am i missing some difference ? (except for passing args)

[TylerReynolds]

@rdeioris I think i understand much more clearly now that you put it that way, being fairly new to both python and unreal engine i'm trying my best to follow along xD
I looked into multithreading and think i understand now how it can be implemented using the threading python module, so i will give it a try.
So everything works including queuing system and thread management with multithreading enabled, right?

I'll leave this issue open for now, so that i can focus on experimenting with the new stuff and seeing how it is working for me, so i'll post once again when i'm done testing to let you know whether or not everything is working properly, thank you very much for your help!

@getnamo Thank you for your reply and suggestion!
Seeing as @rdeioris has some doubts concerning your approach, and considering i'm quite a newbie so i don't understand what the create_and_dispatch thing is, nor why JSON would be needed to convert to structs (not sure i know what structs are), i'll wait for you both to iron out those issues if there are any and will gladly implement anything you both agree with.

Thank you once again for your contribution!

[getnamo]

@rdeioris You are correct, it is using the create_and_dispatch_when_ready taskgraph function in the ue.run_on_gt() with modified code to support passing in arguments. This last part was critical for my use case which is typically long running operations in the background with need to get timely updates on the game thread with data.

As for why I use a renamed version? It's mainly contextual coming from wanting to use similar concepts to android's runOnUiThread e.g.

MainActivity.this.runOnUiThread(new Runnable() {
    public void run() {
        Log.d("UI thread", "I am the UI thread");
    }
});

You may wonder why I do not use the CreateAndDispatchWhenReady function for both threads, while it does support other threads than game thread, the taskgraph system will lock your game if any of their task threads stalls for longer than 3 seconds. This makes the function inappropriate for long running background tasks or loops. So I used a python thread for convenience, which was less verbose than using an FRunnable.

Sadly python doesn't support proper lambdas, but by using inline defined python functions, the event driven style comes through with ut.run_on_bt() => do some heavy work, have a result? => ue.run_on_gt() which makes things performant, safe to access in UE, and easy to write in python. But that's an opinionated approach and the reason why I haven't opened a pull request as it would only make sense if others like that approach to multithreading too :).

[TylerReynolds]

Okay, finally had couple minutes today to experiment with multi-threading, and looks like it worked like a charm right away!

Only problem is, when i close the game the background thread still stays open, and keeps going even if i close the project itself.

I know this is supposed to happen because i didn't really define when to close it, but since i need it running all the time whilst i'm playing i think i'll have to define something to close it when the game stops, or actually even better when the project is closed (it's even better not to have to reload the ram every time i stop playing, for example when testing)

So my question now is, how do i make the thread stop when the game stops, or preferably when the project itself is closed?

@getnamo I got a little confused about JSON conversion and stuff, but reading again your post i guess i have nothing to worry about, as i just need to call those two functions since you handled everything in your code?

Also, does your fork solve the problem of stopping the threads when the game/editor is closed?

[rdeioris]

Set the daemon flag as True:

t = threading.Thread(target=foobar)
t.daemon = True
t.start()

[TylerReynolds]

Will this close it when the editor is closed or the game?
I'd like to know how to handle both scenarios, thank you very much for your quick reply

Also, what do you think about getnamo's fork, should i use it?
I have poor judgement here because of lack of substantial knowledge, but he seems to know what he is doing and it looks like a simple thing to set up?

[rdeioris]

Putting a python thread in daemon mode will means that when the main program (the editor in your case) exits, even the daemon thread will be destroyed

[TylerReynolds]

Thank you!
Oh, i see, how would i destroy the thread when the game stops playing instead?
It should be something like the method begin_play(self)?
I mean, i guess there is a method like that one to call when the game is closed?
So that i can then specify to destroy the thread when the game is closed

Also, what about getnamo's fork, do you think i should use it, is it safe to use?

[rdeioris]

you could subscribe to the 'OnEndPlay' event of the actor (never tried it honestly, so feedback will be interesting). Regarding getnamo 's fork i will include its features in the next days

[TylerReynolds]

That's fantastic, thank you, i will definitely provide some feedback and will also try getnamo's fork, so thank you @getnamo for your work!

[getnamo]

@rdeioris
Never knew about that trick about daemons, it makes perfect sense though, cheers for the tip.

Since I mainly use python components, I just added an end_play() signature on the python component which intercepts the game exit and I can gracefully request threads to stop (usually ending training early in tensorflow). But seeing the documentation for events does your automagic event system work for python components too? if so would I just do

def on_end_play(self, reason):
    pass

or self.uobject.bind_event('OnEndPlay', self.manage_endplay) in beginplay?

@TylerReynolds
Generally it's recommended to gracefully stop threads via e.g. having them run in a loop and check a boolean every now and then. When the boolean (e.g. ShouldRun) gets switched you quit out of your loop and do any cleanup you need so you don't have memory leaks (though this might not be a problem for python so you may may be able to Kill, but it will depend on what you're doing).

Also forget the stuff about I mentioned about JSON, it's only useful for large/generic structured data in generalized used cases (e.g. I need this custom struct data I defined in BP in python). The cost of that is both some performance (JSON conversion) and in my case that another plugin dependency and you want to avoid adding dependencies as much as possible. So if the main repository can provide what you're looking for, definitely stick with it. Looking forward to @rdeioris take on MT feature implementation, one day I may not need to use a fork either :).

[TylerReynolds]

@getnamo I see, but the whole daemon thing should take care automatically of memory leaks and so on, in fact i checked the ram and there was no leakage.
I will try some stuff using your fork because eventually i'm gonna need to work with machine learning and deep learning, just getting warmed up trying things here and there to get familiar with a python ue integration and workflow

@rdeioris Testing the whole virtual machine thing concept, i tried creating a variable in a script and accessing it from another script, but it doesn't seem to be working.
Basically, when the game starts one object creates a variable and another is supposed to read it.

Or does it work only for objects?

---

Actually, i seemed to recall something about an exec function and reading your reply about the virtual machine more thoroughly and about executing the script in the vm i concluded that only scripts executed through the ue.exec('foobar') make their variables available globally.

In fact, using exec the variables were available for reading, awesome.

Is there another way, or is exec the only way to "load" variables and objects globally, so that they can be accessed by any other scripts/objects?

[getnamo]

@TylerReynolds consider wrapping both functions in a class and have the stopping variable be a member variable. So you'd then do if not self.ShouldRun: then stop in the thread loop, and you'd have say another function which just sets the member variable

def setShouldRun(isTrue):
    self.ShouldRun = isTrue

in my case I use the flip version (shouldstop) here https://github.com/getnamo/tensorflow-ue4/blob/master/Content/Scripts/TFPluginAPI.py#L44
which is checked and used to break the training loop in my thread in a specialized example subclass e.g. here
https://github.com/getnamo/tensorflow-ue4-examples/blob/master/Content/Scripts/mnistSimple.py#L81

If you need to access this from another module, make sure you call the function on the same instance to get the desired result.

[TylerReynolds]

Thank you very much @getnamo, but i think the whole virtual machine concept should make things very straightforward since everything is accessible when running in a virtual machine, it's not just a bunch of separated scripts with no connection, so i was looking into this vm thing to take full advantage of it.

But the solution you provided is very interesting when working in a different environment (when executing scripts from a shell like i'm used to), and i also just noticed your tensorflow plugin which i'll definitely have some fun with as soon as i have a more sound knowledge about unreal and this plugin xD

@rdeioris So, i noticed the scripts executed have variables that can be accessed by the console, but not from within a script?
Trying to access the variable coming from the executing script works from the console, but not from within a script.

Should i make the variable global or something?
Or use any particular method?

I don't understand why the console can access them whereas scripts cannot
Something tells me the .exec works only for the console environment, but if so i don't understand how to make variables and objects globally accessible since they are within the same vm the moment they are declared from any script

For the moment my workaround is to simply use a python script that acts as a launcher by calling ue.exec(), and it's working just fine, but i'd like to know what exactly is going on and why scripts cannot access global variables without using the .exec method, is this the only way the virtual machine works?

[TylerReynolds]

Well, everything is working very nicely so far and i understand both uepython and unreal engine much better now, so things have been quite smooth.

One problem i encountered now, though, is about making a thread return a result, so that i can then pass it along in the node "call python actor method string", which calls a python method from a blueprint and returns a string.
I'm quite puzzled by this one, as i would need to use thread.join() to wait for the result the thread would give, but this would mean stopping the main thread and waiting for the result of the thread - making multi-threading completely pointless.

How would i solve such a problem?
@getnamo Since you are working with tensors and advanced machine learning, did you encounter such an issue?
How did you solve it?

---

I realized it's impossible to continue the blueprint execution if the thread is still going and needs time to elaborate the result, so i simply set up a custom event that fires when the thread is done.

Do you think there is a more efficient or better approach?

[getnamo]

@TylerReynolds I handled this by expanding this plugin's taskgraph callback to take arguments (mentioned earlier), and since it uses UE4's threading system there is no need to use .join while still having the results return asynchronously on the game thread.

Regarding the second point, I use the same method, make the ue taskgraph function with the result call a custom bp function, which then forwards the result to a event dispatcher inside my python blueprint component (which is the tensorflow component). That way any other blueprint using my tensorflow component can receive events by clicking the green + next to the OnResults event dispatch.

[rdeioris]

@getnamo finally multithreading is enabled by default and with a solid api. I think the only step missing is the create_and_dispatch_when_ready() variant you made. Would you like to make a pull request ?

[getnamo]

Excellent I'll extract the method and make a pull request in the following week

[getnamo]

pull request made: #564

[ahmadhajmosa]

Dear all

I am trying to use Threading with UE4 and Python, but whenever I start the thread UE4 crashes. Could you please help me in this matter?
My code is as follows:

     def capture_scene(self):
         tic = time.perf_counter()
         self.pawn.call_function('capturescene')

         toc = time.perf_counter()
         ue.log(toc-tic)

    def tick(self, delta_time):
        # get current location

        self.t += delta_time
        if self.t >= 1:
            self.t1 = Thread(target = self.capture_scene)
            self.t1.start()
            ue.log('exccc')
            self.t = 0

I am using UE4 4.19 Mac