WIP: adds gradle build

[christopher-johnson]

I submit this PR with the expectation that it will not be merged into master, but possibly into a development branch. My intent is to make building, testing of apix extensions for integration into upstream platforms (fcrepo,karaf,activemq,etc.) easier by depending on docker and gradle.

It needs a code review, and further analysis of the dependencies as there may be some unnecessary declarations.

The manifests are generated with BND (just like the maven felix plugin) and the gradle plugin reads the blueprints (hence the xerces compiler warnings), but the "Require-Capability" header seems to not work when provisioning in Karaf, so it is filtered. Getting the OSGI wiring to work was a real puzzle, but even after everything resolved, there were several undeclared blueprint feature dependencies (eg. camel-jetty, fcrepo-camel and fcrepo-service-activemq) which were hard to trace. Provisioning the modules as individual bundles is the right way to do OSGI, though with blueprint, it is a bit of a headache as Karaf will provide misleading dependency traces for injected services... Also, the IT tests do not work yet and are filtered in the build. I refactored as little as possible to make it work.

Here is a summary of the code changes:

1. added features.xml definitions for each module and moved to resources.
2. UpdateListener apix does not filter broadcasted messages #138
3. added the optionsEnabled=true here.
4. changed from "sysprop" style configuration variables to camel case for the groovy property expander.
5. removed hardcoded "localhost" from LoaderRoutes and blueprint.
6. build.gradle and gradle.properties
7. removed all pom.xml ... (not necessary, but I am bothered by poms...)

The docker image (0.3.1-SNAPSHOT) is here . Note that this requires my fcrepo and activemq images to work with the core.yml docker-compose file in the root.

[acoburn]

@christopher-johnson I am really supportive of a move toward Gradle. Re: integration testing, there are some tricky things to sort out with this, but in the end Gradle is considerably more flexible than Maven in this regard. If you want to see how I am doing OSGi integration testing (using pax-exam) with other karaf-deployable services in Gradle, take a look here: https://gitlab.amherst.edu/acdc/repository-extension-services/blob/master/acrepo-itests/build.gradle

This uses the gretty plugin to spin up an instance of Fedora and run all the tests.

[birkland]

I don't really know gradle, groovy, jcenter et al.; and am not really sure my mental model of how it all works is even accurate, so it's hard for me to fully understand what this PR is doing, outside of the text description.

That being said, it's clear that there are many parts to this PR that aren't strictly related to a Gradle build. There are some code/configuration cleanups, it looks like the Karaf features file is now hard-coded rather than generated by the features maven plugin, etc.

Would it be possible to break out bits and pieces of this PR in a manner that can be merged into master? It would be great/ideal if this PR were whittled down to just the Gradle bits, with the rest merged into master (or moved elsewhere, like perhaps the docker-compose files; I think we need to revisit the question as to whether any Docker stuff belongs in fcrepo-api-x at all). At that point, we (the API-X stakeholders, and community) can have a serious discussion about whether it makes sense to jump over to Gradle.

[christopher-johnson]

@acoburn @birkland thank you for the feedack.

1. it is definitely possible to separate this into several pieces (fixes, tweaks, and the osgi implementation in gradle).
2. the non-trivial and important part of this is the last piece. The change is that each module is now provisioned as a feature bundle that can be monitored separately. The maven features plugin did not do this and I guess that the whole app is deployed as one über bundle...
3. Docker compose is great for testing with variable build environments. I cannot think of a better tool actually. Dockerfiles are in fact app resources and should definitely be in the repo. Gradle can expand any ${variable} in any file and this makes dynamic testing build config easy and effectively eliminates the need for mvn -D style runtime configs.

I will be in the tech meeting tomorrow.

[birkland]

I don't quite understand this part:

The change is that each module is now provisioned as a feature bundle that can be monitored separately. The maven features plugin did not do this and I guess that the whole app is deployed as one über bundle

It ultimately produces a features file that looks like this:
http://central.maven.org/maven2/org/fcrepo/apix/fcrepo-api-x-karaf/0.1.0/fcrepo-api-x-karaf-0.1.0-features.xml

Individual features can be deployed individually, or one can use an "uber-feature" like feature:install fcrepo-api-x-karaf

not using the features plugin (and instead just manually editing a features file in the repository) is OK too. Neither way is particularly good, but if only one of them works with Gradle, then switching the way feature files are created is fine.

[acoburn]

I have always been unsatisfied with the available tooling for generating features.xml files. The maven-features plugin is pretty good in certain circumstances, though even with Maven projects, I have usually ended up manually managing the features.xml file. And if Maven support for creating features files is less than ideal, Gradle support is (AFAICT) non-existent.

One thing to remember though is that once Java9 is in use, you're going to have to explicitly manage requires and exports declarations (assuming you're not just going to rely on automatic modules). Presumably, the tooling for Jigsaw modularity will be in place eventually (in this regard, Gradle is ahead of Maven, but it's still not fully "there").

[ajs6f]

The hope is that because of Jigsaw's limited ambit (it's really mostly for the footprint of the Java API itself), it won't be hard to tool automation to inspect the code and develop basic graphs. 🤞

[christopher-johnson]

@birkland I did not try to use the maven features plugin, but you are right, it does it correctly (for the most part). I was not aware that the features.xml in fcrepo-api-x-karaf is not the published version. I do not know why the maven tool adds "project bundle interdependencies" when they are actually not needed. For example, with fcrepo-api-x-routing. fcrepo-api-x-routing-dependencies.txt I compare here the differences in the output of bundle:diag on a deployed module and the declarations in the features.xml.

One alternative to feature:install, is to use obr:start. I guess like features, OBR "should" resolve automatically. Yet, here is an example: obr:deploy-unsatisified requirements.txt... Why does not this not pull in the missing bundles from central? Edit: I guess that all requirements must exist as capabilities in the OBR for package resolution to occur. Here is some info about karaf features vs. OBR.

Populating a Apache Cave repo is basic.
cave:repository-populate apix http://repo1.maven.org/maven2/org/fcrepo/apix/ then add the repo url to OBR. obr:url-add http://localhost:8181/cave/http/apix-repository.xml

Anyway, I think it is important to know about these dependency graphs, though I would definitely prefer to have a tool just do it automatically! I may try a plugin to do features.xml in Gradle. Still not automatic, but better than manually editing XML!

@acoburn thank you for the suggestion about the gretty plugin. I think that for migrating any Spring test/boot or PaxExam dependent JUnit tests (like the ones here or in fcrepo) to Gradle, Gretty is a good choice.

An alternative possibility is to build the service node as a docker container which is then deployed to a (kubernetes driven) IT test farm, where CI could process a bunch of API calls. The fact that the app is deployed in OSGI is actually irrelevant to the IT tests, imho. What is being tested is whether or not the service API(s) return the expected HTTP responses, (and for performance, the periodicity ibid) and this could/should be abstracted from the deployment implementation as much as possible.

Regarding the usefulness of PaxExam, I guess that if the MANIFEST.MF says, " Import Package: a, b and c", then the class interactions with a, b and c can be tested with OSGI feature install testing as you do here. This is a nice part of OSGI, if something is wired incorrectly, it will not deploy, and you know immediately if it works or not.

But, what happens when a,b and c have remotely deployed transitive dependencies (e.g. message queueing, cloud services, etc.). Is it possible (or desirable) to test this distributed service environment in a single build context?