Support defining a list of module overrides for Maven/Java modules #81
Comments
|
+1 Thank you. Please reference your approach so that we can align the IDE 'export Java module', this issue and the I understand that they maybe totally unrelated, but they are tied to the user experience of being able to use Java modules without understanding too much of the deeper problems. |
|
@alesj I merged your branch and added support for Using the latest distribution (yes you have to build one), you can test with:
Using the It will fail with: That's as far as I got. I added lots of overrides to make certain imports It seems that a lot of issues have to do with not only missing dependencies but also that they should be shared, even more so for the runtime than for the compiler. Can you help me with these @alesj ? The source code is: module com.redhat.ceylon.compiler.java.test.cmr.modules.bug1100 "1" {
import java.base "7";
import "commons-codec:commons-codec" "1.4";
import "org.apache.camel:camel-core" "2.9.4";
import "org.apache.camel:camel-jetty" "2.9.4";
}import org.apache.camel.impl { DefaultCamelContext }
import org.apache.camel.component.jetty { JettyHttpComponent }
import org.apache.camel.builder { RouteBuilder }
import java.lang { Thread { currentThread } }
"Run the module `simple`."
shared void run() {
print("Start Camel");
value context = DefaultCamelContext();
context.addComponent("jetty", JettyHttpComponent());
object routeBuilder extends RouteBuilder() {
shared actual void configure() {
from("jetty:http://localhost:8080").log("got request");
}
}
context.addRoutes(routeBuilder);
context.start();
currentThread().sleep(10000);
print("Stop Camel");
}
|
|
Also, I had lots of issues when importing maven modules with a
We wanted that behaviour anyways but thought of adding an annotation to the import for that, but since not using the |
|
Guys using version numbers in your run time imports is a very very bad idea, unless you want to see lots and lots of LinkageErrors like above and/or enter OSGi hell. I know it seems clever but it's going to break down on you really fast as the number of installed modules increases. It would make much more sense to keep a single (or perhaps two or at most three) parallel version streams which you keep up to date via whatever mechanism, and perhaps add in version constraints to make sure things are properly enforced. Depending on specific versions is going to give you nothing but trouble. |
|
The LinkageError itself is due to inheriting from more than one version of the same base class or interface at the same time, which is a direct expression of this problem. |
|
I don't know what you're telling us. I'm pretty sure I only import a single version of camel. I don't know which module is having multiple versions loaded at the same time and which versions these are, and who imported them and why they are even visible to each other. Or how to solve this. This is using nothing but Maven modules, and they do specify versions. Should we attempt to merge them all to one? Is there a way in JBoss Modules to dump a list of loaded modules and their dependencies? |
|
@dmlloyd Not sure what you refer to when mentioning "version streams"? Or the problem with depending on specific versions. Isn't this what JBoss Modules does? Isn't this what slots are for? |
I know there is some JMX support in it. |
|
No, slots are definitely not for supporting N versions of one JAR. The JVM simply is not capable of doing this. Even Maven resolves down to a single cohesive version set (or tries to anyway). And, I'm not the one telling you; the JVM is telling you specifically that you are linking to two versions of one class name from a subclass which is disallowed by the JVM linking rules. It even tells you where those classes came from. Yes the latest versions of JBoss Modules have a dependency printer but iirc it only supports the built-in static module loader. It wouldn't be too hard to write one for your own module loader though. |
No, I meant when you say "using version numbers in your run time imports is a very very bad idea". But isn't that what those versions numbers are for, to be able to decide what goes with what and maybe say "sorry, you're trying to use N different versions of this module, no can do" |
|
Yes, if that's all you're using them for (a.k.a. "constraints"). It's when multiple versions of things get pulled in that it's a problem. Note that this happens not only with versions but also with equivalent JARs with different GAV coordinates. This is why we do not directly use Maven information within the Wildfly appserver. |
I have an issue report in my web browser I've been meaning to submit for a couple of days, but let me say it here too: that error message is, in 2013, totally inadequate. JBoss Modules should be capable of a much better error message. This kind of thing was barely acceptable back in the Clinton administration, but not any more. ;-) |
|
JBoss Modules can only provide what the JVM gives us access to, which is precious little. |
|
@dmlloyd Well, let's debate that somewhere else. |
|
So, @alesj WDYT? It seems to me that if we want to merge every imported Maven module so that only one can live at the same time with a single version, we have to stop doing all this lazy loading of modules because we need the full graph resolved at startup and all dependencies figured out for Maven. So now we have overrides and I'm still not convinced this got us closer to supporting Maven modules. I'm going to guess this isn't an issue for Ceylon modules because the shared imports are actually enforced so two a single module can't ever see two versions of the same module loaded at once, even if it happens privately. |
|
@FroMage I think overrides should be fine, you just need to do it properly aka more exact. |
|
Well, did you find the proper override file to make this work? I doubt most humans will be able to. Especially when it "just works" using Maven and a flat classpath. |
|
@FroMage how did you create the overrides.xml in the first place? |
|
Tic-toc tic-toc… |
|
I created it by trial and error looking at the compiler logs first and then the jboss modules logs for every time it failed and why. |
|
Well, atm it looks you only got the javax.servlet thing wrong -- from 10.000ft. :-) And yes, I do agree it's a non-human task, hence we'll probably need to add some tooling to it asap. |
|
So why don't you try it and see if you can find the proper override? |
|
This is going to slip to 1.1 |
|
@FroMage you asked for feedback about this feature, now I think that I can give my humble opinion about this. Well, I'm a bit pessimistic actually. For instance this is the Dropwizard dependency tree generated by gradle It does not matter if the dependencies here are conceptually correct or not. It just works in the java world. First, you see that gradle (like Maven) use a strategy to remove the version conflicts (2.1.2 -> 2.1.4 means 2.1.2 requested, but 2.1.4 retained). OK, so I guess that a tool could help to solve the Problem is, there are classes of problem that, I think, can only be resolved manually. We have a very good example here. At some point trying to make my Dropwizard work, I got : Actually to make this work the tree must be transformed to This can't be automatized. BTW, I have a better understanding now of the reason why I was able to make my exemple work by importing the Maven modules into Ceylon modules via gradles scripts (in my local Ceylon repos). Bottom line, given that the existing Maven librairies were not created to be used in a non-flat classpath, I'm naively wodering if the classloading scheme of Ceylon should not aggregate the whole Maven dependency tree into a huge virtual 'Maven' Ceylon module (of course such module would share every single class :/ ) This is super ugly but this might be the price to pay to have a real easy interop with Maven. And anyway, it does not prevent people from creating clean Ceylon modules for these jars, with clean dependencies manually created. My two cents. |
|
IMHO Maven overrides are too simplistic for real-life scenarios, and there may not be any further options under module resolver. We have had the conversation a few times and it appears that it would be better handled by 3rd-party tools and the Ceylon ecosystem and let the core Dev team tackle the bigger issues.. Please take a look here: After 1.1, release ceylon maven plugin and ceylon.build tasks may provide more useful interaction with Maven: I don't have a solution yet, but I am trying to tackle the problem from both ends: Ceylon and Maven. |
I agree.
I don't know. The Gradle team has a nice programatic solution for this. For the Ceylon compiler/runtime it might work with a Java/Ceylon class plugin or Ceylon code in the module descriptor.
Yes. I agree too. But it is a pity that the existing Maven interop infrastructure embedded in Ceylon can not be leveraged at least for 80% of the use cases (and the difficult 20% could indeed be handled by the 3rd-party tools)
Yes, I really like the approach you choose. And indeed it will definitly help to solve the 20% hard cases related to Maven interop per se. And this risk is worrying, because for every new Maven library this risk exists. And best case, I may succed, but it does not work out of the box and I can plan several hours (days ?) of trial and error to integrate this new Maven library. That's why I was interested with the modules with multiple resource-root
Great, I'm looking forward to it. |
|
Isn't these problems something that assemblies would solve? Indeed, isn't this |
👍 +1 |
Well, you can't say at the same time that it's an assembly descriptor and that it's rubbish. It is indeed a way to declare overrides for bogus Maven dependencies, which, I gather from guessing is part of your secret project for assemblies. It's XML because that was the easiest thing to try, and insofar as it's part of the assembly functionality, it doesn't seem to be working as expected, because figuring out how to fix Maven dependencies is not trivial, as we've seen. It is, however a good way to test our ideas, and where assemblies will go and where they will face issues. It has been suggested several times that we should put all of Maven dependencies inside a flat classpath, but I've rejected this approach so far because the notion of flat classpath is not well-defined when it comes to modularity. Consider two Ceylon modules which include two distinct Maven modules, which in turn import common Maven modules with different versions. Do we have one "flat classpath group" for each Ceylon module? Or a single one which unifies the two imports? ATM we do dependency resolution on a per-import basis, so we can't unify the two groups after the fact. We could change this, but then when we support dynamic loading of modules, how are we going to unify that third Maven module when the two first are already loaded and resolved? The problem gets worse if the Maven modules import Ceylon modules, because suddenly that flat classpath part can view classes from different versions of the same Ceylon module, which only works in a non-flat classpath. I suppose that we can make the Maven imports much closer to how Maven does it by unifying all the imports and merging the constraints to generate the correct override list. Either with a new command that would produce an It's clear we have issues and we can improve. I suggest we improve one bit at a time to find the right solution. It's not clear at all that a flat classpath is a better solution. It may be, and yes it would solve the part about missing imports, but it would leave the issue about version override resolution, which we have to solve in both cases, and it would open questions about composability. If anyone has other ideas, we're taking. |
|
Moving again. |
We should try to fix all the Maven issues by allowing the user to specify a file which lists a number of Maven overrides which allow us for each module name / version pair to:
The text was updated successfully, but these errors were encountered: