Resolve competing web module war definitions

Date	July 7 2020	Contacts	Jody Garnett
Status	proposed	Release	3.12
Resources		Ticket #	pending
Source code	pending
Funding	GeoCat, GeoNetwork Enterprise and professional services

Overview

The web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability.

This proposal refactors web module structure to resolve this issue.

This proposal should simplify the build experience for everyone, and preserve the developer workflow you are used to, and document the result so it is easier to understand where to make any changes.

Technical Details:

Initially proposed as a simple change to take advantage public OSGeo repository; this change has shown several limitations of the core-geonetwork build system for schema-plugins:

Use of schema-plugin version 3.7 across different branches of core-geonetwork, combined with transitive dependencies, ensure conflicts across active branches of core-geonetwork.
web app build copies contents folder to folder, rather than make use of schema plugin artifact zip and jar.
- The web app war and jetty configurations actually directly watch some folders across module boundaries (sidestepping the distinction between copying folder content or unpacking zip contents). This duplication gives some uncertainty when a change is noticed, and how to reliability pick up a change.
web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability
See "baseline" heading further down this page for a detailed review of the current build system

The above limitations result in an unstable build environment.

References:

Proposal: Refactor into distinct modules and src folders

This approach:

jars managed as maven dependencies (to prevent conflict between jetty and libs folder)
- Defines the war using src, target and maven dependencies for jars
- Expect jetty:run to make use of above war configuration above
Any required content is staged into a distinct target folder by process-resources
The war and jetty configuration avoids the use of folders across modules or directories.
- Discussion: An argument can be made for jetty to make use of folders across module boundaries to avoid the need for process-resources
Reserver target/geonetwork for the default use of war packaging (and jetty:run-war testing) and never use it directly

Discussion:

Consider use of web context to replace use of filters, have multiple jetty-context.xml files to test different dev, prod, inspire
The gn database is created in the current directory, can this move to target folder?
Is it desirable to keep data geonetwork and gn database? Avoid resetting when using mvn clean, or make a reset.

migrate

migrate module: a new module to package database migration

src/main/java
src/main/resources

web

web module:

src/assembly - custom war assembly to include license and avoid accidentally packaging contents
src/main/filters - site specific presets used to modify src/main/webapp contents
src/main/java
src/main/resources
src/main/webapp
src/main/data
src/main/webResources
test/java/org/fao/geonet/Start.java
test/resources/jetty-context.xml
target/classes
target/doc
data_directory/data/ - working data directory for jetty run
target/schemas - schema plugins updated by mvn process-resources
target/generated-webapp - generated webapp contents

Proposal: Refactor and setup distinct war and jetty staging

This approach:

Focuses on locking down the war definition using unpack to extract distinct dependencies zip artifacts managed by maven (with no copying between folders).
- Use of distinct target folders will be similar layout to previous proposal.
- Use of a custom war artifact, if required, to produce exactly what we want.
- There is an opportunity to use jetty:run-war to test the war build chain. While this would respond to mvn process-resources, it would not be intended for interactive development.
Allow the jetty configuration to work across folders
- In this workflow web-ui resources, and schema plugins folders would be used directly, no process-resources required.
- Care taken to only use the maven dependencies to take advantage of jetty ability to watch the maven reactor and notice mvn install being used.
- If this approach works using jetty:run would be quicker and just focused on rapid development, there would not be any need to use process-resources (just update files directly for html, javascript and xslt, or use mvn install for java files).

Discussion:

This proposal has the advantage of clearly supporting the two distinct workflows enjoyed by geonetwork developers. It should represent a slight improvement for developers as process-resources would no longer be required to pick up changes. However there is a duplication of effort, and there may be some frustration as a result. If this frustration is too great the result would be unmaintainable.

web

The package-resources and compile stage use:

src/main/filters
src/main/java
src/main/resources
src/main/webapp
src/main/data
src/main/webResources
target/classes
target/doc - copied docs
target/schemas - schema plugins updated by mvn process-resources
target/webGenerated - generated webapp contents from filters and webResources

The war configuration uses:

src/assembly - custom war assembly to include license and avoid accidentally packaging contents
src/main/resources
src/main/webapp
src/main/data
target/classes
target/doc
target/schemas - schema plugins updated by mvn process-resources
target/webGenerated
target/geonetwork/WEB-INF/lib/ - gn-web jar and dependencies staged here

The jetty configuration uses:

src/main/resources
src/main/webapp
test/java/org/fao/geonet/Start.java
test/resources/jetty-context.xml - these dev settings (assume inspire or prod not needed here)
test/webGenerated - generated webapp contents from filters/dev-env.properties and webResources
target/classes
data_directory/data/ - created from src/main/data for use by jetty
../web-ui/src/main/resources - direct use of web-ui files
../schemas/../**/plugin/ - direct use of schema plugins folders

Baseline web/pom.xml build process

The web module has quite a complex build chain supporting:

war creation of a geonetwork.war that includes schema plugins, a default data directory, and some environment specific customizations
jetty:run supporting something like:
- immediate recognition of changes to html files and javascript files
- delayed recognition of changes to XSLT files (using mvn process-resources)
- creation of gn h2 database in the web folder
- direct use of data directory in src/main/webapp

Background on Build Life Cycle

For this discussion keep in mind the maven build life cycle (common steps in italic, default war stages described):

validate
initialize
generate-sources
process-sources
generate-resources
process-resources - resources:resource goal
compile - compiler:compile goal
process-classes
generate-test-sources
process-test-sources
generate-test-resources
process-test-resources - resources:testResources goal
test-compile - compiler:testCompile goal
process-test-classes
test - surefire:test goal
prepare-package
package - war:war goal
pre-integration-test
integration-test
post-integration-test
verify
install - install:install goal
deploy - deploy:deploy goal

Each pom packaging type (pom, jar, war, ...) registers maven plugins to specific stages of the maven life cycle, along with appropriate plugin configuration. As an example war plugin configuration defines a default assembly definition that includes src/webapp.

References:

src folders

The war packaging for the module defines several src folders:

src/main/java

Define Geonetwork and other classes (compiled into target/classes)
src/main/resources

Define resources (compiled into target/classes)
src/webapp

Static outline of war contents.
- conversion
- doc
- htmlCache
- images
- loc - translations
- resources
- WEB-INF
  - web.xml
- xml
- xsl
- xslt
- geonetwork.css
- modalbox.css

The build generate

src/main/filters

Used to process webResources into target/webapp
src/webResources
- WEB-INF
Content processed into target/webapp

The generate-sources registers two additional src folders:

src/main/java - add-src folder containing Geonetwork class
src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate - add-src folder for sql migration

process-resources

The process-resources stages content into:

src/main/webapp/WEB-INF/data/config/schema_plugins

Collected from schema plugin content
src/webapp/doc - from documentation manuals
target/webapp

Copy filter src/main/webResource based on env configuration

compile

The compile stage builds:

target/classes - from java and resources

package

The maven-war-plugin generates jar:

target/geonetwork/WEB-INF/lib/web-app-3.11.0-SNAPSHOT.jar

Containing:
- src/main/java compiled classes
- src/main/resources contents
- src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate compiled classes
  
  This is unusual having java source code in src/webapp!

The maven-war-plugin generates war structure into target/geonetwork:

catalog
conversion
doc
htmlCache
images
loc
META-INF
resources
WEB-INF
xml
xsl
xslt
modalbox.css
geonetwork.css

Where content is collected from:

target/geonetwork: src/main/webapp files
target/geonetwork: target/webapp files that have been processed
target/geonetwork: ../web-ui/src/main/resources files
target/geonetwork/WEB-INF/data/config/schema_plugins: ../schemas files from each plugin folder
target/geonetwork/WEB-INF/lib/: maven dependencies
with many excludes ...
- xml/schemas/
- WEB-INF/data/*.db
- WEB-INF/data/index/**
- ...
These appear to be a safety measure to avoid including files produced when src/main/webapp is used as a live directory by jetty below.

Observations:

⚠️ The handling of sql is very unusual
The processing (using filters) of webResources appears to duplicate functionality that should be controlled by web container.
- Some work put into making the application configurable should remove the need to make custom wars for different environments.
- If needed for local testing a jetty context for each of these configurations would be a clean approach
⚠️ Working with the data directory in src/main/webapp results in lots of strange workarounds:
- Excludes in the war definition to avoid packaging "running" content
- Not quite documented process to "reset" your environment if you want a clean start
⚠️ : war definition including content form web-ui and schema plugins directly, while process-resource includes the same content

      <plugin>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
          <archive>
            <manifest>
              <addDefaultImplementationEntries>true</addDefaultImplementationEntries>
            </manifest>
            <manifestEntries>
              <Implementation-Build>${git.commit.id}</Implementation-Build>
            </manifestEntries>
          </archive>
          <archiveClasses>true</archiveClasses>
          <!-- Filter resources according to current profile properties
              (see src/main/filters) and copy them to the webapp -->
          <webResources>
            <resource>
              <directory>${project.basedir}/../schemas</directory>
              <includes>
                <include>**/src/main/plugin/*</include>
              </includes>
              <targetPath>WEB-INF/data/config/schema_plugins</targetPath>
            </resource>
            <resource>
              <directory>${project.basedir}/../web-ui/src/main/resources</directory>
            </resource>
            <resource>
              <directory>${build.webapp.resources}</directory>
            </resource>
          </webResources>
          <!-- <packagingExcludes>WEB-INF/data/**</packagingExcludes> -->
          <packagingExcludes>
            xml/schemas/**,
            WEB-INF/data/*.db,
            WEB-INF/data/index/**,
            WEB-INF/data/data/backup/**,
            WEB-INF/data/data/resources/htmlcache/**,
            catalog/lib/style/bootstrap/docs/**,
            catalog/lib/style/bootstrap/fonts/**,
            catalog/lib/style/bootstrap/grunt/**,
            catalog/lib/style/bootstrap/test-infra/**,
            catalog/lib/style/font-awesome/css/**,
            catalog/lib/style/font-awesome/src/**,
            catalog/lib/style/font-awesome/scss/**,
            catalog/lib/style/font-awesome/*.json,
            catalog/lib/style/font-awesome/Gemfile*,
            catalog/lib/style/font-awesome/.*,
            catalog/lib/style/font-awesome/*.txt,
            catalog/lib/style/font-awesome/*.md,
            catalog/lib/style/bootstrap/*.json,
            catalog/lib/style/bootstrap/Gemfile*,
            catalog/lib/style/bootstrap/.*,
            catalog/lib/style/bootstrap/*.txt,
            catalog/lib/style/bootstrap/*.md,
            catalog/lib/bootstrap-table/*.json,
            catalog/lib/bootstrap-table/Gemfile*,
            catalog/lib/bootstrap-table/.*,
            catalog/lib/bootstrap-table/*.txt,
            catalog/lib/bootstrap-table/*.md,
            catalog/lib/bootstrap-table/docs/**
          </packagingExcludes>
          <!--          <warSourceDirectory>src/main/geonetwork</warSourceDirectory> -->
          <webXml>${build.webapp.resources}/WEB-INF/web.xml</webXml>
          <attachClasses>true</attachClasses>
          <warName>${application.name}</warName>
          <webappDirectory>${project.build.directory}/geonetwork</webappDirectory>
        </configuration>
      </plugin>

jetty

The jetty-maven-plugin defines web application using:

maven dependencies
target/geonetwork
src/main/webapp
../web-ui/src/main/resources/
target/webapp
jett-context.xml: used to prevent jetty scanning jars during startup

Observations:

This configuration has been setup to use the "live" src/main/webapp location and pick up any changes each time mvn process-resources is called above.
⚠️ schemas plugin folders are included twice:
- in target/geonetwork via war definition
- in src/main/webapp/ via process-resource copy
⚠️ jars are included twice:
- target/geonetwork libs folder
- as maven dependency
- example: An error starting up (where domain jar has a conflict between jar included in libs and included by maven)

      <plugin>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-maven-plugin</artifactId>
        <configuration>
          <contextXml>${basedir}/jetty-context.xml</contextXml>
          <webAppSourceDirectory>${project.build.directory}/geonetwork</webAppSourceDirectory>
          <webApp>
            <contextPath>/${application.name}</contextPath>
            <descriptor>${project.build.directory}/WEB-INF/web.xml</descriptor>
            <baseResource implementation="org.eclipse.jetty.util.resource.ResourceCollection">
              <resourcesAsCSV>
                ${project.basedir}/src/main/webapp,
                ${rootProjectDir}/web-ui/src/main/resources/,
                ${build.webapp.resources}
              </resourcesAsCSV>
            </baseResource>
          </webApp>
          <systemProperties>
            <systemProperty>
              <name>org.eclipse.jetty.server.Request.maxFormContentSize</name>
              <value>5000000</value>
            </systemProperty>
          </systemProperties>
          <httpConnector>
            <port>${jetty.port}</port>
          </httpConnector>
          <stopKey>JETTY_TOP</stopKey>
          <stopPort>${jetty.stop.port}</stopPort>
        </configuration>
      </plugin>