Extending the supported data types

[arjenmarkus]

I have used this library to interact with numPy from a simple Tcl program, to see whether this would work and what would be required for a convenient API. I ran into a number of issues, some of which have already been solved, but I would appreciate advice on a number of things: like what is an efficient way to pass numerical data to and fro?

[gwlester]

Arjen,

This library seems rather dead -- maybe try to get DKF to take a look and interest. He has done a decent amount of Python also.

[arjenmarkus]

Hi Gerald, I intend (and Steve Landers does too) to talk about this in the online conference. Pity if it is indeed dead. Regards, Arjen Op ma 19 okt. 2020 om 16:56 schreef Gerald Leter <[email protected]>:

…

Arjen, This library seems rather dead -- maybe try to get DKF to take a look and interest. He has done a decent amount of Python also. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR7RGTL5JFWDDAYYDTDSLRHQLANCNFSM4SV3RMSA> .

[aidanhs]

Hi @arjenmarkus

First off, regarding death, I'll be honest - I don't use Tcl these days so don't plan to do any more work on this repo. I'm happy to review and merge PRs (or add contributors), or provide guidance to interested parties on ways to go about things - the codebase is tiny, under 600 lines of mostly error handling. Alternatively I would be open to talking about contracting - sadly there are only so many hours in the day for hobby work!

To answer your second question: it really depends. Generally speaking the best way to make FFI across language boundaries fast (particularly for high level languages like Python and Tcl) is to move data as little over the boundary as possible.

So, with a more complete libtclpy and numpy pseudocode (I've not really used it), you'd do something like

py eval {
    def do_things(num_list, logline):
        print(logline)
        arr = numpy.array_from_list(num_list)
        arr.reverse()
        return arr.to_python_list()
}
set mylist [list 1 2 3]
set out [py call -typed do_things "list<number>" $mylist string "hello"]
puts $out

(note: the -typed argument doesn't exist today, so everything is passed as a string, which is bad - there is no decent workaround, you'll need to use JSON or something, it'll be slow)

If you didn't want to do it in one shot (allowing you to call functions freely later on), you'd take the classic approach for Tcl C extensions and return a handle, like:

py eval {
    def make_array(num_list):
        return tclpy.Handle(numpy.array_from_list(num_list))
}
set mylist [list 1 2 3]
set arr [py call -typed make_array "list<number>" $mylist]
puts $arr; # something like "tclpy0x123456abcd"
py destroy $arr

(note: handles don't exist today - a workaround would be to create a python function stashing the array in globals() with a randomly generated name and use that as the handle)

However, if you just want numpy then you probably want to create a set of bindings around the numpy C bindings itself. This is a hassle, as you're just recreating bindings that already exist in Python, but chops out one hop of the Tcl -> Python -> numpy dance if you really want speed.

[1] Example handle stashing (untested):

def to_handle(array):
    handle = str(uuid.create())
    globals()[handle] = array
    return handle
def from_handle(handle):
    return globals()[handle]

[arjenmarkus]

Thanks for this analysis, My current plan is to use the package "as is" to see where improvements are needed/desirable. Of course, when I (or somebody) get to use it on actual large problems, efficient data management becomes really important. That would be an interesting situation :). One of the attractive aspects of the package is that the code size is indeed so limited. Op do 22 okt. 2020 om 00:05 schreef Aidan Hobson Sayers < [email protected]>:

…

Hi @arjenmarkus <https://github.com/arjenmarkus> First off, regarding death, I'll be honest - I don't use Tcl these days so don't plan to do any more work on this repo. I'm happy to review and merge PRs (or add contributors), or provide guidance to interested parties on ways to go about things - the codebase is tiny, under 600 lines of mostly error handling. Alternatively I would be open to talking about contracting - sadly there are only so many hours in the day for hobby work! To answer your second question: it really depends. Generally speaking the best way to make FFI across language boundaries fast (particularly for high level languages like Python and Tcl) is to move data as little over the boundary as possible. So, with a more complete libtclpy and numpy pseudocode (I've not really used it), you'd do something like py eval { def do_things(num_list, logline): print(logline) arr = numpy.array_from_list(num_list) arr.reverse() return arr.to_python_list() } set mylist [list 1 2 3] set out [py call -typed do_things "list<number>" $mylist string "hello"] puts $out (note: the -typed argument doesn't exist today, so everything is passed as a string, which is bad - there is no decent workaround, you'll need to use JSON or something, it'll be slow) If you didn't want to do it in one shot (allowing you to call functions freely later on), you'd take the classic approach for Tcl C extensions and return a handle, like: py eval { def make_array(num_list): return tclpy.Handle(numpy.array_from_list(num_list)) } set mylist [list 1 2 3] set arr [py call -typed make_array "list<number>" $mylist] puts $arr; # something like "tclpy0x123456abcd" py destroy $arr (note: handles don't exist today - a workaround would be to create a python function stashing the array in globals() with a randomly generated name and use that as the handle) However, if you *just* want numpy then you probably want to create a set of bindings around the numpy C bindings itself. This is a hassle, as you're just recreating bindings that already exist in Python, but chops out one hop of the Tcl -> Python -> numpy dance if you really want speed. [1] Example handle stashing (untested): def to_handle(array): handle = str(uuid.create()) globals()[handle] = array return handle def from_handle(handle): return globals()[handle] — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR44OUUG5ODA7D2SEEDSL5LKRANCNFSM4SV3RMSA> .

[aidanhs]

FYI I've just bumped the library to support (only) Python 3, since Python 2 is now dead

[arjenmarkus]

Great, I will have a look asap. Currently preparing a presentation on this for the online Tcl conference.