.bsp, .bloop, .metals and other friends

[adpi2]

More often than not Metals users have the target, the .bsp, the .bloop and the .metals folders in their working directory. While I understand the purpose of each one of these, I also think there is something wrong in "polluting" the users directory with so much technical folders.

As a visual example I created a simple sbt project with only one module and this is what I have in Metals:

I would like to open this discussion to see if we can improve the situation. Maybe a similar discussion already exists somewhere but I was not able to find it. Also this discussion is not specific to Metals but it needs to start somewhere.

Proposal

When starting a project the users choose a build tool and a code editor. So they will not be surprised that there is a target folder for the build tool and a .metals folder for the code editor. However it is harder to understand or justify that there is a .bsp and a .bloop folder, since they do not result directly from a conscious choice of the user. The newcomers are confused by them as they don't really know which tool creates them for which reasons. They can try to remove them only to see that are coming back at some point.

.bsp and .bloop are more like implementation details of how Metals or sbt works together. For that reason I would argue that the information contained in those two folders should be distributed in the .metals and the target directory. Here is a proposal of how it can be done:

.bsp

The files in .bsp are written by build tools so that code editors can find them.

On one hand it is very easy for the code editor to find those files because there are all in the same place: the .bsp folder. On the other hand it degrades the user experience of many users.

I propose we distribute the .bsp/* files in the build tools target directory. For instance there would be one bsp.json file in the target folder of an sbt repository, and similarly one bsp.json file in the out folder of a mill repository.

The main drawback is that code editors will have to go over all folders to find all the bsp.json files, which would rarely be higher than 100 folders. The cost would not be so high and the result would be a better experience for all users, in particular for the newcomers.

.bloop

In the majority of cases, it is Metals that triggers the generation of the .bloop config for its own purpose and no other tool will ever use it. There are some exceptions though that I treat below.

So I propose that Metals put the .bloop folder inside the .metals folder. This is somehow similar to what scala-cli does: it creates a .bloop folder inside the .scala folder and uses Bloop internally to compile.

In terms of implementation we can add a configuration key for the output folder (.metals instead of .) in sbt-bloop, maven-bloop and gradle-bloop.

To deal with the exceptions, there should be a configuration in Metals so that a user can still have the .bloop folder in the root folder. Also Metals should use the existing .bloop folder if there is one already (as it already does).

[ckipp01]

Thanks for starting this discussion @adpi2! I've thought about this a bit in the past as well and more recently seeing how scala-cli does this since they do eliminate the .bloop/ dir. I want to add a few more things to your points.

Regarding .bsp/

One of the other main drawbacks of what you outline is that you sort of throw the idea of a generic BSP discovery out the window. Right now a huge strength of BSP is that in theory a tool can work with a build server having no idea any of the specifics about that build server (or build tool) as long as it can find the .bsp/<entry>.json and therefore know how to start it. scala-cli is actually a great example of this where without Metals knowing anything about scala-cli it will work for the user. With the approach where it would put the bsp entry in it's own .scala-build/ you then lose the ability for a tool to work with it unless it knows specific details about the output dir of the build tool. The same can be said for sbt and target/ and mill and out/. We also do have custom BSP servers out there, not a lot, but some. For these users, this would be a breaking change that would force Metals, IntelliJ, etc to now have to know details of that tool just in order to find the bsp entry.

I'm curious if a global configuration approach has ever been considered for this instead. I wonder if it'd be impossible to instead have some sort of $HOME/.bsp that could keep a mapping of workspaces or something like that. So when a tool opens a workspace it can check that gobal table to see if there is a bsp entry for that workspace? There are probably drawbacks to this as well, but I think having a generic discovery where tools don't need to know the implementation of how or where the bsp entry gets created is a huge strength of BSP.

Regarding .bloop/

This entire one just relates to the future of Bloop and Bloop CLI. I'd love to see usage statistics on Bloop CLI. I have a gut feeling that the Bloop CLI is very rarely used by people. Especially as other BSP servers become more popular, I think the usages is even less. For example I use to use Bloop CLI all of the time. It was my go to. However knowI rarely ever use due to DAP becoming more relevant, and because I use sbt as my BSP server more and more. I know that as you outline as well, this is what scala-cli does and I have seen a very small grumbling about users not being able to use Bloop CLI with it, but I think again that the number is incredibly small. In my opinion, this is the easiest one to get rid of.

[tgodzik]

I do agree with Chris in regards to .bsp, I think it really needs to be in the main directory so that it serves it's purpose.

For .bloop we could hide it in .metals and add an option in BloopCLI to try and search some well know directories such as .metals. I would try to have Bloop CLI working as it did so that people who use it in their workflow are still able to do so.

[adpi2]

Thanks for the replies.

@ckipp01 said

Right now a huge strength of BSP is that in theory a tool can work with a build server having no idea any of the specifics about that build server (or build tool) as long as it can find the .bsp/.json and therefore know how to start it.

That we should definitely keep. But still the spec could be ​<any folder>/<entry>.bsp.json is a BSP connection file. So the build client (Metals) still doesn't need to know anything about the build servers but it needs to scan every folder in the root directory (only one level deep, not recursively). That should not take so long since there should be less than 100 folders.

[tgodzik]

We could do that for sure, but it would require changes in a lot of places (Intellij, Metals, mill, sbt, scalacli) and require someone to drive that effort, which I am not sure gives us that much benefit. I also think having separate .bsp directory is a nice way to indicate that you are able to use BSP here.