Tree sitter integration is not triggering

[theothornhill]

Hi there!

I've been trying to get this to work for some time now, but I cannot for the life of me get it to work. I'm using an M1 mac, so that might be an issue, but I don't think so.

(tree-sitter-available-p) returns t, and i've tried to move the dylibs around in different folders to get emacs to pick up on them. I've been following your RFC page and also the manuals, but no dice.

Non working recipe:

1. Compile casouri/emacs on ts branch, reports that it finds tree-sitter >= 0.0
2. Run batch-new.sh and move the files in dists to /usr/local/bin (among others)
3. Launch emacs from src/emacs
4. (require 'tree-sitter)
5. Open some C file
6. M-x ts-c-mode

All syntax highlighting dissapears, and tree-sitter-inspect-mode shows nil. I assume this is because it cannot find the dylibs, but I'm stumped atm. Could you kindly give me some pointers as to what I'm missing? I'm sorry for the probably very stupid mistake I've made :)

[00-matt]

Hi, I have the same issue!

json-mode seems to work fine, turning on tree-sitter-inspect-mode describes everything as expected.

ts-c-mode does not work at all, there is no syntax highlighting, and trying to indent gives the message tree-sitter-indent: Wrong type argument: tree-sitter-node-p, nil.

Unfamiliar with elisp and Emacs internals so I'm not sure how to debug this further, I have /usr/lib64/libtree-sitter-c.so which is a symlink to /usr/lib64/libtree-sittitter-c.so.13 (which exists).

edit M-: (tree-sitter-get-parser-create 'tree-sitter-c) seems to return a parser without error.

[casouri]

I'm very sorry guys! GitHub didn't seem to notify me of any issues created on this repo... As for ts-c-mode, I pushed some changes recently, could you give the latest version a try? Thanks!

[00-matt]

Thanks for looking into this! I just rebuilt with commit e11fd97.

I then launch the newly built Emacs and run M-: (require 'tree-sitter). I tried activating ts-c-mode in the scratch buffer by running M-x ts-c-mode but there is no highlighting, running M-x tree-sitter-inspect-mode shows "N/A "N/A"" in the modeline. It seems like something related to json-mode has stopped working too as I get the same behaviour.

It seems like the grammar files are not actually being loaded. Running awk '/tree-sitter/ { print $6 }' /proc/$(pgrep emacs)/maps to inspect the loaded shared libraries shows that libtree-sitter.so.0.0 has loaded in, but not libtree-sitter-c.so.13.

[casouri]

Could you try (tree-sitter-parser-create nil 'tree-sitter-c) and see if it raises an error? Also check the *Messages* buffer and see if logs any error during redisplay.

[00-matt]

After evaluating (tree-sitter-parser-create (current-buffer) 'tree-sitter-c), #<tree-sitter-parser for tree-sitter-c in #<buffer *scratch*>> shows up in the *Messages* buffer and then /proc/.../maps shows that libtree-sitter-c.so has been loaded.

[casouri]

Could you try enabling ts-c-mode and see what's printed in *Messages* buffer? If anything goes wrong you should see something like

Error during redisplay: ...

and we'll know what's the problem.

[00-matt]

Thank you for helping me debug.

I don't see any message like that unfortunately :(

[casouri]

It seems like there is no parser in that buffer. Could you show me the value of tree-sitter-parser-list in that buffer? Also tree-sitter-font-lock-settings and tree-sitter-font-lock-defaults. Thanks.

[00-matt]

• tree-sitter-parser-list is nil
• tree-sitter-font-lock-defaults is ((ts-c-tree-sitter-settings-1))
• tree-sitter-font-lock-settings is

((tree-sitter-c ((null) @font-lock-constant-face (true) @font-lock-constant-face (false) @font-lock-constant-face (comment) @font-lock-comment-face (system_lib_string) @ts-c-fontify-system-lib (unary_expression operator: _ @font-lock-negation-char-face) (string_literal) @font-lock-string-face (char_literal) @font-lock-string-face (function_definition declarator: (identifier) @font-lock-function-name-face) (declaration declarator: (identifier) @font-lock-function-name-face) (function_declarator declarator: (identifier) @font-lock-function-name-face) (init_declarator declarator: (identifier) @font-lock-variable-name-face) (parameter_declaration declarator: (identifier) @font-lock-variable-name-face) (preproc_def name: (identifier) @font-lock-variable-name-face) (enumerator name: (identifier) @font-lock-variable-name-face) (field_identifier) @font-lock-variable-name-face (parameter_list (parameter_declaration (identifier) @font-lock-variable-name-face)) (pointer_declarator declarator: (identifier) @font-lock-variable-name-face) (array_declarator declarator: (identifier) @font-lock-variable-name-face) (preproc_function_def name: (identifier) @font-lock-variable-name-face parameters: (preproc_params (identifier) @font-lock-variable-name-face)) (type_identifier) @font-lock-type-face (primitive_type) @font-lock-type-face "auto" @font-lock-keyword-face "break" @font-lock-keyword-face "case" @font-lock-keyword-face "const" @font-lock-keyword-face "continue" @font-lock-keyword-face "default" @font-lock-keyword-face "do" @font-lock-keyword-face "else" @font-lock-keyword-face "enum" @font-lock-keyword-face "extern" @font-lock-keyword-face "for" @font-lock-keyword-face "goto" @font-lock-keyword-face "if" @font-lock-keyword-face "register" @font-lock-keyword-face "return" @font-lock-keyword-face "sizeof" @font-lock-keyword-face "static" @font-lock-keyword-face "struct" @font-lock-keyword-face "switch" @font-lock-keyword-face "typedef" @font-lock-keyword-face "union" @font-lock-keyword-face "volatile" @font-lock-keyword-face "while" @font-lock-keyword-face "long" @font-lock-type-face "short" @font-lock-type-face "signed" @font-lock-type-face "unsigned" @font-lock-type-face "#include" @font-lock-preprocessor-face "#define" @font-lock-preprocessor-face "#ifdef" @font-lock-preprocessor-face "#ifndef" @font-lock-preprocessor-face "#endif" @font-lock-preprocessor-face "#else" @font-lock-preprocessor-face "#elif" @font-lock-preprocessor-face)))

I ran M-: (setq tree-sitter-parser-list (tree-sitter-parser-create (current-buffer) 'tree-sitter-c)) and then pressing return in the C buffer (causing indent to be called?), crashing Emacs with this backtrace:

hread 1 "emacs" received signal SIGBUS, Bus error.
0x000055555579dcc4 in ts_record_change ()
(gdb) bt
#0  0x000055555579dcc4 in ts_record_change ()
#1  0x00005555556ba94d in insert_1_both.part ()
#2  0x00005555556bb014 in insert_and_inherit ()
#3  0x00005555556ce323 in internal_self_insert ()
#4  0x00005555556cea06 in Fself_insert_command ()
#5  0x0000555555710203 in Ffuncall ()
#6  0x000055555574edb2 in exec_byte_code ()
#7  0x000055555571012f in Ffuncall ()
#8  0x000055555570c541 in Ffuncall_interactively ()
#9  0x0000555555710203 in Ffuncall ()
#10 0x000055555570ce80 in Fcall_interactively ()
#11 0x0000555555710203 in Ffuncall ()
#12 0x000055555574edb2 in exec_byte_code ()
#13 0x000055555571012f in Ffuncall ()
#14 0x00005555557102ba in call1 ()
#15 0x000055555569f3f6 in command_loop_1 ()
#16 0x000055555570f037 in internal_condition_case ()
#17 0x000055555568fafe in command_loop_2 ()
#18 0x000055555570ef91 in internal_catch ()
#19 0x000055555568fa89 in command_loop ()
#20 0x000055555569565c in recursive_edit_1 ()
#21 0x00005555556959a4 in Frecursive_edit ()
#22 0x00005555555a51e3 in main ()

I will try and rebuild with debug symbols and no optimisations later so that I can be more helpful :)

[casouri]

Yup, it crashes my Emacs too. Although what you evaluated is "illegal", that definitely shouldn't crash Emacs. I'll try to add some protection against it. On the reason why it is "illegal": tree-sitter-parser-list is meant to be managed by Emacs itself, changing it makes bad things happen. So this crash is most probably not related to the problem we are trying to solve, and probably you don't need to rebuild with debug symbols (yay).

Could you try

(progn
  (ts-c-mode)
  (tree-sitter-font-lock-fontify-region (point-min) (point-max)))

on a fresh buffer with toggle-debug-on-error on? Thanks. Right now I'm not sure what went wrong, since all the values you showed me are perfectly good.

[00-matt]

Oh, that's good to know!

Here is the backtrace after evaluating the progn snippet:

Debugger entered--Lisp error: (error "Font-lock trying to use keywords before setting them up")
  signal(error ("Font-lock trying to use keywords before setting them up"))
  error("Font-lock trying to use keywords before setting them up")
  font-lock-compile-keywords(nil)
  font-lock-fontify-keywords-region(1 16 nil)
  font-lock-default-fontify-region(1 16 nil)
  tree-sitter-font-lock-fontify-region(1 16)
  (progn (ts-c-mode) (tree-sitter-font-lock-fontify-region (point-min) (point-max)))
  eval((progn (ts-c-mode) (tree-sitter-font-lock-fontify-region (point-min) (point-max))) t)
  eval-expression((progn (ts-c-mode) (tree-sitter-font-lock-fontify-region (point-min) (point-max))) nil nil 127)
  funcall-interactively(eval-expression (progn (ts-c-mode) (tree-sitter-font-lock-fontify-region (point-min) (point-max))) nil nil 127)
  call-interactively(eval-expression nil nil)
  command-execute(eval-expression)

[casouri]

Could you try again with

(progn
  (ts-c-mode)
  (font-lock-fontify-region (point-min) (point-max)))

? Thanks.

[00-matt]

Looks like it highlighted the buffer! tree-sitter-inspect-mode shows the node type, and indenting using the TAB key also works.

However pressing RET to start a new line with electric-indent-mode, I get this error:

Debugger entered--Lisp error: (wrong-type-argument tree-sitter-node-p nil)
  tree-sitter-node-parser(nil)
  tree-sitter-indent()
  indent-according-to-mode()
  electric-indent-post-self-insert-function()
  self-insert-command(1)
  newline(nil 1)
  funcall-interactively(newline nil 1)
  call-interactively(newline nil nil)
  command-execute(newline)

Also if I add more text, it doesn't get highlighted until I rerun font-lock-fontify-region (point-min) (point-max).

[casouri]

I pushed a fix, could you give it a try?

[00-matt]

It's working great for me now! Just enabling ts-c-mode gives me indentation and syntax highlighting.

Thank you for your time.

[theothornhill]

I finally found some time to test this as well now, and it seems to work reliably now. It seems the blocker concerning dynamic loading in tree-sitter has been merged. What is left for this to make it into mainline emacs? I'd love to start hacking on the major modes I maintain :)

[casouri]

I've sent the patch to Emacs-devel. They haven't come around to it yet. Maybe after Emacs 28 is released?

[theothornhill]

Hi there, @casouri and thanks for getting back to me by mail. I'll take it here because that tree-sitter thread gets very bikesheddy very fast :)

I've started implementing tree-sitter for this implementation over at https://github.com/emacs-typescript/typescript.el/blob/feature/tsx-support/typescript-ts.el, and it is going pretty well actually. Kudos!

However, I cannot for the life of me understand how the indentation is supposed to work, even though I've read the manual a hundred times by now. Can you help me get started with the indentation?

For example, I'd like this to work for starters:

function foo() {
  | <-- point should be indented to spaces
}

Same with:

function foo() {
  try {
    | <-- point should be here
  }
}

My guess is that the first one is matching the no-node rule, and the try should match to some parent of sort, but I cannot get anything to work. I'm also having trouble with the ts-c-mode. Pressing tab doesn't seem to do anything, and using the verbose mode doesn't either.

What am I missing?

[casouri]

Hey Theo, sorry that I missed your message. Judging from your latest message on emacs-devel that I just read, you already fixed or at least understand this problem?

[theothornhill]

Yeah, I believe so. I think I'm about 90% done with the typescript mode, actually. I've also made some changes after that message pushing me even a little further, so that's good :)