New to scala: cannot get autocomplete or intellisense to work

[baslat]

Hi,

I need some help getting metals to work properly please. At the moment I cannot even get intellisense or autocomplete to work. I'm pnew to scala, coming from an R (and to a lesser extent python) background, so there might be some very obvious things that I don't know and don't know that I don't know. I cannot stress enough how little I know about java and scala and spark.

I use VS Code and metals and scala as a REPL to work on spark on an EMR cluster via spark-shell. So far I haven't need to compile things (and I don't really know the process to compile things) VS Code gives me syntax highlighting, but that's about it. So for example I make a file called script.scala that looks like this:

import org.apache.spark.sql.{DataFrame, SaveMode}
import java.text.SimpleDateFormat
import org.apache.commons.lang.time.DateUtils

spark.read.

and I expect the metals language server to start offering autocompletion, after spark.read. but it doesn't. If it were an R script, called script.R and looking like

library(dplyr)
library(lubridate)
library(readr)

readr::read_

the R langauge server would start offering intellisense and help documentation and function automplete from those packages.

I feel like I am missing something fundamental in the scala/java/spark/metals ecosystem.

Please help as I am trying to learn scala and spark but am really struggling to harness the power of metals. Thanks!

[tgodzik]

Thanks for the question! script.scala are normal Scala files, which require using toplevel objects for example:

object Main extends App{
  ... // statements
}

I see there are some nice docs here https://spark.apache.org/docs/latest/sql-data-sources-load-save-functions.html you can probably find how the different languages work with the same code.

If you'd like to work with a script, you can create script.sc file, which is the extension used for Scala scripts. Not sure if you are using a build tool, but if not you might want to check out ScalaCLI. We are actually working at making it very Spark friendly, it's an experimental feature but the docs can be found here https://scala-cli.virtuslab.org/spark/

[baslat]

Ok thanks, that's helpful. I started a coursera course on scala and it mentioned the .scala and .sc filetypes, and showed that the .sc calculated results automatically and showed them in a similar manner to comments. However I wasn't sure how this would work with spark on a cluster, and I didn't want it to try to read and process scores of 50GB files on the fly, so I went for the .scala, which might have gone against what metals expects given the scripty-nature of my code.

I'll check out the Scala CLI build tool. Is there a beginner's guide that explains the relationships between metals, scala scripts, scala files, build tools and spark?

[tgodzik]

Is there a beginner's guide that explains the relationships between metals, scala scripts, scala files, build tools and spark?

Not currently, but there is a plenty of documentation that you can take a look at. I did some talks about Metals that might be helpful, but not exactly about the 5 things you mention. There is an introduction here https://youtu.be/iATmBDxlYAA, but you should be able to find more videos on the topic with metals scala search.

[baslat]

I've managed to get some minor success, but I don't understand why. I have a file called test.sc with the contents

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf

val ten = 4+6

I made a build.sbt file with the contents

name := "my-spark-project"

version := "0.1.0"

scalaVersion := "2.12.15"

libraryDependencies += "org.apache.spark" %% "spark-core" % "3.2.0"

I get these errors:

But if I comment them out, it works:

But even more weirdly, if I move test.sc to src/main/scala/test.sc, then it works

Please explain what is happening here. Why is folder structure so critically important to that one error, but not the rest of the evaluation? And I don't really know what I did or why it worked, or if all the changes were necessary. Again, just trying to learn, thanks!

[tgodzik]

We add the classpath (dependencies etc.) to the worksheet (*.worksheet.sc) when it's located within the source directories of the project. For usual sbt projects it's src/main/scala so spark is only added if the worksheet is located there. If the spark dependency is not added then the file fails to compile due to the imports.

You can also add dependencies using special syntax within the worksheet and that should also work:

import $ivy.`org.apache.spark::spark-core:3.2.0` 

I recommend reaching out on the scala discord https://discord.com/invite/scala where we have a dedicated channel for Spark and people there can suggest best ways to work with Spark. I unfortunately am no expert in that regards, so I might be of a lesser help.