Resolve competing web module war definitions

Date	July 7 2020	Contacts	Jody Garnett
Status	proposed	Release	3.12
Resources		Ticket #	pending
Source code	pending
Funding	GeoCat, GeoNetwork Enterprise and professional services

Overview

The web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability.

This proposal refactors web module structure to resolve this issue.

Technical Details:

Initially proposed as a simple change to take advantage public OSGeo repository; this change has shown several limitations of the core-geonetwork build system for schema-plugins:

Use of schema-plugin version 3.7 across different branches of core-geonetwork, combined with transitive dependencies, ensure conflicts across active branches of core-geonetwork.
web app build copies contents folder to folder, rather than make use of schema plugin artifact zip and jar.
web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability

The above limitations result in an unstable build environment.

References:

Background

For this discussion keep in mind the maven build life cycle (common steps in italic, default war stages described):

validate
initialize
generate-sources
process-sources
generate-resources
process-resources - resources:resource goal
compile - compiler:compile goal
process-classes
generate-test-sources
process-test-sources
generate-test-resources
process-test-resources - resources:testResources goal
test-compile - compiler:testCompile goal
process-test-classes
test - surefire:test goal
prepare-package
package - war:war goal
pre-integration-test
integration-test
post-integration-test
verify
install - install:install goal
deploy - deploy:deploy goal

Each pom packaging type (pom, jar, war, ...) registers maven plugins to specific stages of the maven life cycle, along with appropriate plugin configuration. As an example war plugin configuration defines a default assembly definition that includes src/webapp.

References:

Proposed build process

migrate

migrate: package database migration in a separate module

src/main/java
src/main/resources

web

web:

src/assembly - custom war assembly to include license and avoid accidentally packaging contents
src/main/filters - site specific presets used to modify src/main/webapp contents
src/main/java
src/main/resources
src/main/webapp
src/main/data
src/webapp
test/java/org/fao/geonet/Start.java
test/resources/jetty-context.xml
target/classes
target/doc
target/data/
target/data/config/schema_plugins
target/schemas - schema plugins updated by mvn process-resources
target/webapp - generated webapp contents

This approach:

jars managed as maven dependencies
Defines the war using src, target and maven dependencies for jars
Expect jetty:run to make use of above war configuration above
Any required content is staged into a target folder by process-resources
Any war and jetty configuration avoids the use of folders across modules or directories.
Reserver target/geonetwork for the default use of war packaging (and jetty:run-war testing)

Discussion:

Use of web context to replace filters, have multiple jetty-context.xml files to test different dev, prod, inspire
The gn database is created in the current directory, can this move to target folder?

Baseline web/pom.xml build process

The web module has quite a complex build chain.

src folders

The war packaging for the module defines several src folders:

src/main/java

Define Geonetwork and other classes (compiled into target/classes)
src/main/resources

Define resources (compiled into target/classes)
src/webapp

Static outline of war contents.
- conversion
- doc
- htmlCache
- images
- loc - translations
- resources
- WEB-INF
  - web.xml
- xml
- xsl
- xslt
- geonetwork.css
- modalbox.css

The build generate

src/main/filters

Used to process webResources into target/webapp
src/webResources
- WEB-INF
Content processed into target/webapp

The generate-sources registers two additional src folders:

src/main/java - add-src folder containing Geonetwork class
src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate - add-src folder for sql migration

process-resources

The process-resources stages content into:

src/main/webapp/WEB-INF/data/config/schema_plugins

Collected from schema plugin content
src/webapp/doc - from documentation manuals
target/webapp

Copy filter src/main/webResource based on env configuration

compile

The compile stage builds:

target/classes - from java and resources

package

The maven-war-plugin generates jar:

target/geonetwork/WEB-INF/lib/web-app-3.11.0-SNAPSHOT.jar

Containing:
- src/main/java compiled classes
- src/main/resources contents
- src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate compiled classes
  
  This is unusual having java source code in src/webapp!

The maven-war-plugin generates war structure into target/geonetwork:

catalog
conversion
doc
htmlCache
images
loc
META-INF
resources
WEB-INF
xml
xsl
xslt
modalbox.css
geonetwork.css

Where content is collected from:

target/geonetwork: src/main/webapp files
target/geonetwork: target/webapp files that have been processed
target/geonetwork: ../web-ui/src/main/resources files
target/geonetwork/WEB-INF/data/config/schema_plugins: ../schemas files from each plugin folder
target/geonetwork/WEB-INF/lib/: maven dependencies
with many excludes ...
- xml/schemas/
- WEB-INF/data/*.db
- WEB-INF/data/index/**
- ...
These appear to be a safety measure to avoid including files produced when src/main/webapp is used as a live directory by jetty below.

      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
            <manifestEntries>
              <Implementation-Build>${git.commit.id}</Implementation-Build>
            </manifestEntries>
          </archive>
          <archiveClasses>true</archiveClasses>
          <!-- Filter resources according to current profile properties
              (see src/main/filters) and copy them to the webapp -->
          <webResources>
            <resource>
              <directory>${project.basedir}/../schemas</directory>
              <includes>
                <include>**/src/main/plugin/*</include>
              </includes>
              <targetPath>WEB-INF/data/config/schema_plugins</targetPath>
            </resource>
            <resource>
              <directory>${project.basedir}/../web-ui/src/main/resources</directory>
            </resource>
            <resource>
              <directory>${build.webapp.resources}</directory>
            </resource>
          </webResources>
          <!-- <packagingExcludes>WEB-INF/data/**</packagingExcludes> -->
          <packagingExcludes>
            xml/schemas/**,
            WEB-INF/data/*.db,
            WEB-INF/data/index/**,
            WEB-INF/data/data/backup/**,
            WEB-INF/data/data/resources/htmlcache/**,
            catalog/lib/style/bootstrap/docs/**,
            catalog/lib/style/bootstrap/fonts/**,
            catalog/lib/style/bootstrap/grunt/**,
            catalog/lib/style/bootstrap/test-infra/**,
            catalog/lib/style/font-awesome/css/**,
            catalog/lib/style/font-awesome/src/**,
            catalog/lib/style/font-awesome/scss/**,
            catalog/lib/style/font-awesome/*.json,
            catalog/lib/style/font-awesome/Gemfile*,
            catalog/lib/style/font-awesome/.*,
            catalog/lib/style/font-awesome/*.txt,
            catalog/lib/style/font-awesome/*.md,
            catalog/lib/style/bootstrap/*.json,
            catalog/lib/style/bootstrap/Gemfile*,
            catalog/lib/style/bootstrap/.*,
            catalog/lib/style/bootstrap/*.txt,
            catalog/lib/style/bootstrap/*.md,
            catalog/lib/bootstrap-table/*.json,
            catalog/lib/bootstrap-table/Gemfile*,
            catalog/lib/bootstrap-table/.*,
            catalog/lib/bootstrap-table/*.txt,
            catalog/lib/bootstrap-table/*.md,
            catalog/lib/bootstrap-table/docs/**
          </packagingExcludes>
          <!--          <warSourceDirectory>src/main/geonetwork</warSourceDirectory> -->
          <webXml>${build.webapp.resources}/WEB-INF/web.xml</webXml>
          <attachClasses>true</attachClasses>
          <warName>${application.name}</warName>
          <webappDirectory>${project.build.directory}/geonetwork</webappDirectory>
        </configuration>
      </plugin>

jetty

The jetty-maven-plugin defines web application using:

maven dependencies
target/geonetwork
src/main/webapp
../web-ui/src/main/resources/
target/webapp
jett-context.xml: used to prevent jetty scanning jars during startup

Observations:

This configuration has been setup to use the "live" src/main/webapp location and pick up any changes each time mvn process-resources is called above.
schemas plugin folders are included twice:
- in target/geonetwork via war definition
- in src/main/webapp/ via process-resource copy
jars are included twice:
- target/geonetwork libs folder
- as maven dependency

      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <configuration>
          <contextXml>${basedir}/jetty-context.xml</contextXml>
          <webAppSourceDirectory>${project.build.directory}/geonetwork</webAppSourceDirectory>
          <webApp>
            <contextPath>/${application.name}</contextPath>
            <descriptor>${project.build.directory}/WEB-INF/web.xml</descriptor>
            <baseResource implementation="org.eclipse.jetty.util.resource.ResourceCollection">
              <resourcesAsCSV>
                ${project.basedir}/src/main/webapp,
                ${rootProjectDir}/web-ui/src/main/resources/,
                ${build.webapp.resources}
              </resourcesAsCSV>
            </baseResource>
          </webApp>
          <systemProperties>
            <systemProperty>
              <name>org.eclipse.jetty.server.Request.maxFormContentSize</name>
              <value>5000000</value>
            </systemProperty>
          </systemProperties>
          <httpConnector>
            <port>${jetty.port}</port>
          </httpConnector>
          <stopKey>JETTY_TOP</stopKey>
          <stopPort>${jetty.stop.port}</stopPort>
        </configuration>
      </plugin>