Clarify the need of scala-reflect and quasiquote dependency jars

[jrudolph]

The macro documentation says this:

Macros needs scala-reflect.jar in library dependencies.

This says a bit too little about when exactly the dependency is needed. In the three stages of

1. macro compilation (compilation classpath)
2. macro expansion (compilation classpath)
3. user code runtime (JVM classpath)

not every stage will need scala-reflect on the classpath. Similarly for quasiquotes.

Two related issues:

A) What dependency is needed at what stage?

So far, I thought that scala-reflect only needs to be on the compiler classpath during 1. while quasiquotes during 1. and 2. (because of the QuasiquoteCompat "expansion time" component).

So, for Scala 2.10 this matrix seems to work:

Stage	Classpath type	scala-reflect	quasiquotes	macro module
Macro compilation	compiler classpath	yes	yes	-
Macro expansion	compiler classpath	no	yes	yes
Runtime	JVM classpath	no	no	no

Does that sound reasonable?

I now got these kinds of errors when compiling on Scala 2.11.x:

scala.ScalaReflectionException: object scala.reflect.macros.blackbox.Context in compiler mirror not found.
  at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:123)
  at scala.reflect.internal.Mirrors$RootsBase.staticClass(Mirrors.scala:22)
  at akka.parboiled2.support.OpTreeContext$Action$$typecreator28$1.apply(OpTreeContext.scala:558)
  at ...

during 2. which seems to suggest that Context type somehow leaked to the next stage (may be a parboiled2 issue).

So, the matrix seems to be this for 2.11.x:

Stage	Classpath type	scala-reflect	macro module
Macro compilation	compiler classpath	yes	-
Macro expansion	compiler classpath	yes	yes
Runtime	JVM classpath	no	no

Do you have an idea why that would be the case?

B) How do you declare these dependencies

The next issue is how to get these dependencies properly configured in the build system.

The goal is that no library is on the classpath that isn't necessary at this stage, most importantly you don't want any compile time artifacts leak to final runtime.

The usual setup is to just declare "compile" scope dependencies for scala-reflect and quasiquotes in the macro project and for the macro module in the user-code. This defeats the goal as all modules turn up as hard dependencies at runtime.

The only solution is a manual one where for each "yes" above you need to add a manual dependency that is scoped "provided" to prevent leakage to the next stage. This means that as a macro library publisher you cannot "just ship" your library but you have to remind your users also to add a provided dependency on quasiquotes.

Besides some macros like parboiled2 have compile-time and runtime and components. It would be cool if you could just declare one dependency and have the dependency management system figure out what the compile-time artifact is and what the runtime artifact.

This is no urgent issue. In fact, if you can accept that those compile-time artifacts leak to runtime, it is no issue at all. However, for library writers that either publish or use macros it is currently hard to do the-right-thing-tm, so I thought I write it up and post it here.

[jrudolph]

Maybe some background: In spray/akka@ba09096 I made the exception disappear also in Scala 2.11 by moving a typeOf[Rule[_, _]] statement out of a pattern guard.

[xeno-by]

A) The matrix is correct. If there's a deviation, that's either a bug in scalac or an unfortunate situation in a particular macro. What's there at OpTreeContext.scala:558 in akka.parboiled2?

[xeno-by]

B) "This means that as a macro library publisher you cannot "just ship" your library but you have to remind your users also to add a provided dependency on quasiquotes". Yes, I agree that this is a problem for the QuasiquoteCompat module in 2.10. Does it manifest itself on 2.11 as well?

[xeno-by]

"It would be cool if you could just declare one dependency and have the dependency management system figure out what the compile-time artifact is and what the runtime artifact".

Do you have any ideas how to implement the "figure out" part? Could there be a non-heuristic algorithm that does that?

Also, another question here is how to express that knowledge in the form of maven metadata. I've chatted with sbt guys about that back then and got an answer that we have to use Ivy for that, and this looks pretty marginal.

[jrudolph]

@xeno-by thanks for the answer. I guess the main question is if it is really worthwhile to look hard enough for solutions. By "figure out" I thought of an Ivy solution but one would have to experiment a bit with it.

What's there at OpTreeContext.scala:558 in akka.parboiled2?

It's just a typeOf used in a pattern guard. If you look at the commit linked above moving out the typeOf fixed the problem.

[xeno-by]

@jrudolph Yeah, looks like an unfortunate side effect of how reify works (reify is used to implement typeOf internally). Since you're reifying an existential type, reify creates symbols to represent the existentially quantified identifiers and, as it's currently implemented, it needs an owner chain for those symbols and that's where a reference to Context leaks into. Moving the typeOf should've indeed fixed the problem.

[jrudolph]

I read again through this ticket (in relation to com-lihaoyi/upickle#116) and I didn't understand any more why scala-reflect would not be on the classpath at "Macro expansion" time, i.e. when a macro is run during compilation. To explain it again to me and future readers: of course, scala-reflect is in the runtime class path of the compiler and, thus, a macro implementation can call into scala-reflect methods. However, what you cannot do is expect to have scala-reflect on the compilation classpath (which totally makes sense). So, you cannot lookup symbols from scala-reflect, etc, i.e. reflect on scala-reflect in the macro.

The bug here is, as Eugene explained in the previous comment, that some typeOf-like operations may close over the syntactical macro environment and accidentally reify reflection accesses to scala-reflect types. So, I guess a similar thing is happening in com-lihaoyi/upickle#116.

[xeno-by]

@jrudolph Would you consider contributing to our documentation at https://github.com/scala/scala.github.com/tree/master/overviews/macros with the things that you've learned about the topic at hand?

[retronym]

I've create a minimal reproduction of this problem: https://gist.github.com/retronym/1159dd01f2e1c0b26a3601030bfd2579