Wezterm Mux Server causes weird line artifacts in neovim

[miversen33]

What Operating System(s) are you seeing this problem on?

Linux X11

Which Wayland compositor or X11 Window manager(s) are you using?

KDE Plasma WM

So this is a weird one. After updating to the most recent nightly, I have noticed that while using neovim, I will often see lines being blank, in the wrong place, shifting around as I move the cursor throughout a file, etc.

I was able to verify this is a Wezterm issue and not a neovim issue by using neovim in several different terminals with the same neovim configuration. I have tested this in the following terminals

• Alacritty
• Konsole
• Wezterm (no mux server)

In all cases, neovim performs as expected. However if I use a unix domain, suddenly I see weird artifacts when navigating a file. Below is a video of this
https://github.com/wez/wezterm/assets/2640668/ac9ea99a-b05b-443b-9f6a-41e7069a48a8

And here is a video showing me using wezterm with the same file without the mux server.
https://github.com/wez/wezterm/assets/2640668/f297228f-1986-4ab6-8668-a600b5dbeb1f

I have no idea what is going on. Help?

WezTerm version

20231120-164150-fde92672

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

There are line artifacts happening when using neovim within the wezterm mux server

To Reproduce

To make life as easy as possible, below is a bash script that can be used to get you into a similar state as what I was seeing (hopefully)

#!/usr/bin/bash

TEST_DIR=/tmp/wezterm-test
EDIT_FILE=https://raw.githubusercontent.com/wez/wezterm/main/mux/src/tab.rs
EDIT_FILE_NAME=tab.rs
NEOVIM_VERSION=nightly # can also be stable

mkdir -p $TEST_DIR
cat > $TEST_DIR/wezterm.lua << EOF
local wezterm = require("wezterm")
return {
    unix_domains = {
        { name = "wezterm-test-domain" }
    },
}
EOF

cat > $TEST_DIR/neovim-config.lua << EOF
-- Minimal configuration
-- mini.lua
-- Use with the --clean -u flags. EG nvim --clean -u mini.lua
-- This config will create a temp directory and will blow away that temp directory
-- everytime this configuration is loaded. Great for simulating a new installation
-- of a plugin

-- Setting some basic vim options
-- Some junk because I am sick of formatting tables in print
local _print = _G.print
local clean_string = function(...)
    local args = { n = select("#", ...), ... }
    local formatted_args = {}
    for i = 1, args.n do
        local item = select(i, ...)
        if not item then item = 'nil' end
        local t_type = type(item)
        if t_type == 'table' or t_type == 'function' or t_type == 'userdata' then
            item = vim.inspect(item)
        end
        table.insert(formatted_args, item)
    end
    return table.concat(formatted_args, ' ')
end
_G.print = function(...)
    _print(clean_string(...))
end

vim.opt.mouse = 'a'
vim.opt.termguicolors = true
-- If you want to play around with this, you can set the do_clean
-- variable to false. This will allow changes made to
-- underlying plugins to persist between sessions, while
-- still keeping everything in its own directory so
-- as to not affect your existing neovim installation.
--
-- Setting this to true will result in a fresh clone of
-- all modules
local do_clean = false
local sep = vim.loop.os_uname().sysname:lower():match('windows') and '\\\\' or '/' -- \\ for windows, mac and linux both use /
local root = vim.fn.fnamemodify("./.repro", ":p")
if vim.loop.fs_stat(root) and do_clean then
    print("Found previous clean test setup. Cleaning it out")
    -- Clearing out the mods directory and recreating it so
    -- you have a fresh run everytime
    vim.fn.delete(root, 'rf')
end

-- DO NOT change the paths and don't remove the colorscheme

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
    vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({ "git", "clone", "--filter=blob:none", "https://github.com/folke/lazy.nvim.git", lazypath, })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
    "folke/tokyonight.nvim",
    -- add any other plugins here
}

require("lazy").setup(plugins, {
    root = root .. "/plugins",
})

vim.opt.splitright = true
vim.opt.splitbelow = true

vim.cmd.colorscheme("tokyonight")
-- add anything else here
EOF

pushd $TEST_DIR 2>&1 1>/dev/null
if [ ! -d $TEST_DIR/nvim-linux64 ]; then
    echo "Fetching Neovim $NEOVIM_VERSION"
    curl -Ls https://github.com/neovim/neovim/releases/download/$NEOVIM_VERSION/nvim-linux64.tar.gz | tar -xz
fi
if [ ! -f $TEST_DIR/$EDIT_FILE_NAME ]; then
    echo "Fetching File to edit in neovim: $EDIT_FILE"
    curl -Lso $TEST_DIR/$EDIT_FILE_NAME $EDIT_FILE
fi

echo "you might have to run this script twice as the start command fails if the domain isn't running first..."
wezterm --config-file $TEST_DIR/wezterm.lua connect wezterm-test-domain -- bash -c "cd $TEST_DIR && $TEST_DIR/nvim-linux64/bin/nvim --clean -u $TEST_DIR/neovim-config.lua $TEST_DIR/$EDIT_FILE_NAME"
popd 2>&1 1>/dev/null

echo "Created test directory in ${TEST_DIR}. You might wanna clean that up"

Configuration

return {
    unix_domains = {
        { name = "local unix socket"}
    },
    default_gui_startup_args = {
        "connect",
        "local unix socket"
    }
}

Expected Behavior

I would expect my terminal to properly render the lines as they are shown

Logs

22:07:59.752 WARN wezterm_client::client > While connecting to Socket("/run/user/1000/wezterm/sock"): connecting to /run/user/1000/wezterm/sock. Will try spawning the server.
22:07:59.752 WARN wezterm_client::client > Running: "/usr/bin/wezterm-mux-server" "--daemonize"

Anything else?

I notice this kind of artifacting behavior gets even worse if I put splits in my neovim configuration. This all started happening today after an update to the latest wezterm nightly. I don't know what version I was running prior unfortunately :(

[tbung]

I have the same issue with the release version of wezterm. This issue was caused by an update to neovim, I am currently bisecting which commit caused it. I also cannot reproduce the issue with nvim --clean so it has to be some weird interaction between a plugin and some change to neovim core. I'll report back when I find out more and will open an issue with neovim core or the relevant plugin if it turns out that that is warranted.

[tbung]

This issue was introduced by neovim/neovim@ac8ed77. Neovim now supports synchronized output. Apparently, wezterm is supposed to support this. I was wrong about this being related to plugins, they just make it more obvious. So either the neovim implementation of this is wrong, which I don't think since it works outside the mux environment, or there is an issue with this in the mux environment in wezterm.

As a workaround, you can just disable this in neovim: set notermsync or in lua: vim.opt.termsync = false.

PS: Don't update your main editor to the newest nightly and then bisect some breaking change, when you actually have a work deadline coming up. Don't ask me how I know.

[miversen33]

Turning off termsync in my neovim config did fix the issue. I appreciate you :) Hopefully Wez gets around to poking this soon but at least I can use the muxer again :)

[youngtuotuo]

This issue was introduced by neovim/neovim@ac8ed77. Neovim now supports synchronized output. Apparently, wezterm is supposed to support this. I was wrong about this being related to plugins, they just make it more obvious. So either the neovim implementation of this is wrong, which I don't think since it works outside the mux environment, or there is an issue with this in the mux environment in wezterm.

As a workaround, you can just disable this in neovim: set notermsync or in lua: vim.opt.termsync = false.

PS: Don't update your main editor to the newest nightly and then bisect some breaking change, when you actually have a work deadline coming up. Don't ask me how I know.

Same issue here, thx for the workaround.

[wheatdog]

I'm facing the same issue. Unfortunately, the workaround doesn't work for me. I'll provide more information later.

[tbung]

Can confirm, although this time around I did not yet have the time to fully investigate. However what I can confirm is, that the issue turns up even before termsync became a neovim option, I tested all versions from 0.9.0 to nightly on wezterm-nightly and latest release. (Currently only tested on MacOS, will test Linux after work)

Simple steps to reproduce:

• Attach to a SSH Mux session
• Open a file larger than the screen in nvim (nvim --clean test.txt or nvim --clean -c 'lua vim.opt.termsync=false test.txt` for later versions)
• Split the screen :sp
• Go to upper window and scroll down. The lower half of the screen gets corrupted until something updates in the lower half.

The last point also prevented me from noticing this, as something in my plugins keeps the lower half of the screen updated, though I did not yet have the time to bisect my plugins. Maybe someone with more time could either try to figure out what changed so we can disable this again, or even try to figure out how to fix termsync in wezterm mux.

[hadronized]

I have the same problem with Kakoune.

[tbung]

I finally had the time to bisect the issue to d36ad7c, particularly reverting the change here

wezterm/wezterm-mux-server-impl/src/sessionhandler.rs

Lines 103 to 112 in d36ad7c

	.filter_map(|(idx, mut line)| {
	let stable_row = first_line + idx as StableRowIndex;
	if all_dirty_lines.contains(stable_row) {
	all_dirty_lines.remove(stable_row);
	line.compress_for_scrollback();
	Some((stable_row, line))
	} else {
	None
	}
	})

seems to fix the issue. I will take a closer look at this, do some more testing and then submit a pull request in the next few days.

Edit: Changing that part back does not fix the issue completely, but atleast with set notermsync neovim works again. I shall investigate further.

[tbung]

So what I've been seeing in total was lines not getting updated properly and also lines moving around, as in this demo. Note the line numbers in the beginning.

wezterm_bug_demo.mp4

All of these troubles were not that obvious and consistent in my day to day setup but are easily and consistently reproducible with OP's config and nvim --clean -c 'set relativenumber' some_long_file.

Turns out, there is two issues at play here. The regular artifacts (lines not updating properly) are due to a race condition, which also explains why they didn't appear all the time and were way worse with termsync on. This has already been fixed in #5981 (although not yet merged).

The other issue, were lines broke when scrolling in neovim (also tested in regular vim, just to make sure it's not just a neovim issue) was harder to find and I'm not completely sure what set of conditions has to be met for it to be visible, but apparently the viewport moved down and a scrollback built up even when there should not be a scrollback (for example in alternate screen mode). I think I fixed it in #6099.

With both those commits I don't see any glitches anymore.

[tbung]

As far as I can tell this issue can be closed now. Maybe some of the others in this thread also want to confirm.

[hadronized]

I haven’t tried, give me a couple of minutes to build a local version of wezterm and test with Kakoune.

[tbung]

@hadronized you can also just install nightly. Both fixes have been merged.

[hadronized]

Yeah, I don’t see issues anymore. Sounds okay to me!