Is os file monitoring the right approach for detecting file updates?

[eatkins]

I was thinking about the problem of monitoring large sets of files for source updates and making this work reliably (i.e. swoval's raison d'etre). It strikes me that if the programmer is using an LSP enabled editor, then the editor should be able to notify the LSP server whenever it saves a file. This seems better to me than file monitoring most of the time because there are a lot of weird edge cases that crop up with file system events (e.g. does the editor implement save by writing to a temp file and moving or by writing in place?).

I'm imagining a new world where there is an additional metals config option that enables/disables os file monitoring and adds a new endpoint to the LSP server that receives file change events. If file monitoring is disabled, then the client will need to add a hook that notifies the LSP server whenever a source file is saved. My sense is that this would be pretty straightforward in the client plugin for many if not most editors. When file monitoring is enabled, metals would just do exactly what it already does.

Beyond potentially more consistently reliable file event notifications on all platforms, moving away from file monitoring would significantly lower the memory usage for large projects.

One drawback is that if the programmer modifies a file in their project with a different editor that is not connected to the LSP server, then that file could get out of date WRT to metals. This seems relatively uncommon and would be resolved whenever they next save the file in the LSP connected editor anyway or restart metals.

Explicit editor generated file event notifications could be rolled out gradually client by client, so I'd imagine the switch could be done with fairly minimal disruption (i.e. most end users would not ever even notice that the event system changed after they upgraded).

[tgodzik]

Thanks for opening the discussion and also sorry for not responding for so long! As far as LSP goes there is workspace/didChangeWatchedFiles and workspace/didChangeWatchedFolders , more info here https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_didChangeWatchedFiles.

However, this apparently wasn't working too reliably (might have changed since @olafurpg checked it) and besides we do watch a lot of files unrelated to the editor. It might be possible to also watch them if editor supports dynamic registration, but we would need to experiment more about that to see if it's feasible. It might also be impossible in some cases, for example VS Code ignores .bloop directory by default in order not to search it, which might also extend to file watching.

I think the biggest issue is that we actually create so many files unrelated to the editor that is falls far from LSP domain. Id does fall into the BSP one, so maybe we should get file notifications from there? This would mean no file watching is needed as BSP would send everything it created. That would require a ton of work also.

[eatkins]

No problem on the response time, I just found myself musing about swoval/swoval#161 and thought what if it were possible to eliminate file watching entirely while preserving the same dev experience? As the author of a library that does file watching, I know that it is a very difficult problem to get completely right. I believe that swoval is maybe a 99% solution, but getting it to 100% is almost impossible. Even though I get very few bug reports, about once a year or so someone reports a bug/edge case that is difficult to diagnose and/or fix. I assume there are others that haven't been reported. This experience has brought me to the idea that push is a better model than pull for this kind of thing and somewhat easier to reason about, but I also can understand how that might not be possible for the metals use case specifically. There are a lot of design decisions in scala that would probably be different if LSP enabled editors existed 15 years ago.

[tgodzik]

We might try to use VS Code watching as an alternative, maybe it would be plausible to use? 🤔

[eatkins]

I would be somewhat surprised if their implementation was better than swoval. the advantage though is that I will probably EOL swoval at some point (or at least request that the community take it over). I have moved off of scala and the jvm in my personal work so I've been supporting it as a charity for the last few years.

[ritschwumm]

hm - how would i get (command-line) git to push changes when checking out a new branch?

@eatkins i feel your pain - file watching indeed does seem to be impossible to get right. so many people (myself included) tried, and i've yet to see a tool where it doesn't fail from time to time under mysterious circumstances :/

[tgodzik]

hm - how would i get (command-line) git to push changes when checking out a new branch?

That is probably the biggest issue.

[eatkins]

I think this is a theoretically solvable issue, but in a way that might be unpalatable to users. Using a tool like direnv, the git command itself could be shimmed so that write commands like merge, pull, rebase, branch, etc. connect to the language server and issue a file change update for whatever files changed after the operation. The simplified shell code would be like

function git_branch() {
  start_hash=(git rev-parse HEAD)
  git branch $*
  lsp_notify (git diff --name-only $start_hash)
}

Based on my experience with sbt users, I do not think this would be a popular solution, but I do think it could conceivably work pretty well. It seems likely to me that most use cases could potentially be supported without file watching, but it probably isn't compelling enough to implement unless there is a big user who wants to pay for the work to be done because they are frustrated by performance issues like in the referenced swoval issue.

I do think that the push model is conceptually nicer in many ways and there could always be a manual refresh command if things get out of sync. Again though, I'll admit that I am not being totally helpful here because I'm not actually interested in doing the work myself. I'm more motivated to deprecate swoval so that I don't feel obliged to maintain it.

[tgodzik]

Thanks anyway for starting the discussion, I think this is very valuable to start and think about the alternatives. We might not have the time work on that currently, but I will keep that topic in mind.