FFI Design Discussions

I open this page to discuss the design of the FFI system.

Current situation

A primitive function (oget <object> <property1> <property2> ...) is exposed. A base *root* variable designates the main object (which is established by the runtime).

On NodeJS it doesn't work properly though as there are some variables that are in the module scope but are not in global, but I think those are just a few and the NodeJS runtime can provide them as special cases.

On top of that, a #j macrocharacter is implemented so (#j:console:log "test") is read as ((oget *root* "console" "log") "test") which basically does the right thing.

Problem 1: accessing properties of your own CL variables

#j: always work with the root object, which make it inconvenient to work with your own CL variables. You can't do

(defvar canvas (#j:document:querySelector "#canvas"))
(#j:canvas:getContext "2d")  ;  would be read as ((oget *root* "canvas" "getContext") "2d")

Ideas

• Make #j:a:b:c expand to (oget a "b" "c"). Then you can use *root* if you need to or it could be provided by the shortcut #j::a:b (oget root "a" "b"). I don't like the first argument being different but not the syntax. Perhaps its hould be (#j/a:b:c) instead??

Problem 2: Constructing new objects

Problem 3: Conversion between Lisp and JS values

oget converts between JS and Lisp values automatically. Sometimes we do not want this, how should we disable this? Another variant of #j:: or #j/? A special variable *convert-ffi-values*? My concern with the latter is that special variables could not play well with async code.

Problem 4: Define a Lisp-y interface to foreign JS Classes and Objects

While it is currently possible to access individual object properties and methods using oget it would be useful to provide a higher-level abstraction so that foreign JS objects can be accessed as if they were Lisp ADTs. (e.g. structures defined using defstruct and operations on them.)

Providing such a high-level interface has several advantages:

• It saves the programmer from remembering the low-level details of the JS/JSCL bridge because they only need to provide a high-level declarative form of the class interface they want to access to.

• It makes it easier to debug code using these classes by mocking them in a standard CL environment. We still have very limited debugging facilities in JSCL itself.

I have been working on a prototype for a define-js-foreign-class macro, see https://gist.github.com/michipili/07f4ce79d06a59440578df6f67033c14 for a full example.

With this macro the shortened specification of native JS arrays

(define-foreign-js-class (js-array (:validate-class t))
  ((length
    :documentation "The length property of an object which is an instance of type Array sets or returns the number of elements in that array."
    :type number))
  ((push
    :documentation "The PUSH method adds one or more elements to the end of an array and returns the new length of the array.")
   (pop
    :documentation "The POP method removes the last element from an array and returns that element. This method changes the length of the array."))))

Is expanded to

(PROGN
 (DEFUN JS-ARRAY-P (X) T)
 (DEFSETF JS-ARRAY-LENGTH #:SETF-JS-ARRAY-LENGTH734)
 (DEFUN #:SETF-JS-ARRAY-LENGTH734 (INSTANCE-PARAMETER VALUE-PARAMETER)
   (UNLESS (JS-ARRAY-P INSTANCE-PARAMETER)
     (ERROR "The object `~S' is not of type `~S'" INSTANCE-PARAMETER
            "JS-ARRAY"))
   (JSCL:OSET VALUE-PARAMETER INSTANCE-PARAMETER "length"))
 (DEFUN JS-ARRAY-LENGTH (INSTANCE-PARAMETER)
   (UNLESS (JS-ARRAY-P INSTANCE-PARAMETER)
     (ERROR "The object `~S' is not of type `~S'" INSTANCE-PARAMETER
            "JS-ARRAY"))
   (JSCL:OGET INSTANCE-PARAMETER "length"))
 (DEFUN JS-ARRAY-PUSH (INSTANCE-PARAMETER &REST ARGS)
   (UNLESS (JS-ARRAY-P INSTANCE-PARAMETER)
     (ERROR "The object `~S' is not of type `~S'" INSTANCE-PARAMETER
            "JS-ARRAY"))
   (APPLY ((JSCL:OGET INSTANCE-PARAMETER "push" "bind") INSTANCE-PARAMETER)
          ARGS))
 (DEFUN JS-ARRAY-POP (INSTANCE-PARAMETER &REST ARGS)
   (UNLESS (JS-ARRAY-P INSTANCE-PARAMETER)
     (ERROR "The object `~S' is not of type `~S'" INSTANCE-PARAMETER
            "JS-ARRAY"))
   (APPLY ((JSCL:OGET INSTANCE-PARAMETER "pop" "bind") INSTANCE-PARAMETER)
          ARGS)))

It is possible to use the generated bindings to interact with native JS arrays as a stack:

;; In the browser console
;;  var s = [];
;; In JSCL REPL
CL-USER> (defparameter *s* (jscl::%js-vref "s"))
*S*
CL-USER> (js-array-push *s* "Apple" "Pear" "Figue")
3
CL-USER> (js-array-pop *s*)
"Figue"

Problems in the macro code:

• I am not yet able to write the correct code for the predicate recognising the instances of the class we are operating on. See https://gist.github.com/michipili/07f4ce79d06a59440578df6f67033c14#file-define-foreign-js-class-lisp-L100, https://github.com/jscl-project/jscl/issues/296

• It should generate a make-* function calling the constructor.

• When the value returned by oget … "bind" is undefined, the generated error is cryptic (it says that the form starting with oget is not a valid function name or something similar).

• It does not seem possible to used LAMBDAs as the setter expression in short defsetf invocations. Is it as intended? Is there a better way to put the defsetf declaration?

• Using gensyms for instance-parameter and value-parameter yields an error. It seems to be an error in JSCL.

• The :type keyword is only an annotation.

• The :documentation keyword is only an annotation, but it could be used as a documentation string in the defined methods.