Resolve competing web module war definitions

Date	July 7 2020	Contacts	Jody Garnett
Status	proposed	Release	3.12
Resources		Ticket #	pending
Source code	#4701
Funding	GeoCat, GeoNetwork Enterprise and professional services

Overview

The web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability.

This proposal refactors web module structure to resolve this issue.

Technical Details:

Initially proposed as a simple change to take advantage public OSGeo repository; this change has shown several limitations of the core-geonetwork build system for schema-plugins:

Use of schema-plugin version 3.7 across different branches of core-geonetwork, combined with transitive dependencies, ensure conflicts across active branches of core-geonetwork.
web app build copies contents folder to folder, rather than make use of schema plugin artifact zip and jar.
web app contains competing build configurations (for both war:war and jetty:run) on the management of schema-plugins leading to build instability

The above limitations result in an unstable build environment.

Background

For this discussion keep in mind the maven build life cycle (common steps in bold, default war stages described):

validate
initialize
generate-sources
process-sources
generate-resources
process-resources - resources:resource goal
compile - compiler:compile goal
process-classes
generate-test-sources
process-test-sources
generate-test-resources
process-test-resources - resources:testResources goal
test-compile - compiler:testCompile goal
process-test-classes
test - surefire:test goal
prepare-package
package - war:war goal
pre-integration-test
integration-test
post-integration-test
verify
install - install:install goal
deploy - deploy:deploy goal

Each pom packaging type (pom, jar, war, ...) registers maven plugins to specific stages of the maven life cycle, along with appropriate plugin configuration. As an example war plugin configuration defines a default assembly definition that includes src/webapp.

References:

Baseline web/pom.xml build process

The web module has quite a complex build chain.

src folders

The war packaging for the module defines several src folders:

src/main/java

Define Geonetwork and other classes (compiled into target/classes)
src/main/resources

Define resources (compiled into target/classes)
src/webapp

Static outline of war contents.
- conversion
- doc
- htmlCache
- images
- loc - translations
- resources
- WEB-INF
  - web.xml
- xml
- xsl
- xslt
- geonetwork.css
- modalbox.css

The build generate

src/main/filters

Used to process webResources into target/webapp
src/webResources
- WEB-INF
Content processed into target/webapp

The generate-sources registers two additional src folders:

src/main/java - add-src folder containing Geonetwork class
src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate - add-src folder for sql migration

process-resources

The process-resources stages content into:

src/main/webapp/WEB-INF/data/config/schema_plugins

Collected from schema plugin content
src/webapp/doc - from documentation manuals
target/webapp

Copy filter src/main/webResource based on env configuration

compile

The compile stage builds:

target/classes - from java and resources

package

The maven-war-plugin generates jar:

target/geonetwork/WEB-INF/lib/web-app-3.11.0-SNAPSHOT.jar

Containing:
- src/main/java compiled classes
- src/main/resources contents
- src/webapp/main/webapp/WEB-INF/classes/setup/sql/migrate compiled classes
  
  This is unusual having java source code in src/webapp!

The maven-war-plugin generates war structure into target/geonetwork:

catalog
conversion
doc
htmlCache
images
loc
META-INF
resources
WEB-INF
xml
xsl
xslt
modalbox.css
geonetwork.css

Where content is collected from:

target/geonetwork: src/main/webapp files
target/geonetwork: target/webapp files that have been processed
target/geonetwork: ../web-ui/src/main/resources files
target/geonetwork/WEB-INF/data/config/schema_plugins: ../schemas files from each plugin folder
target/geonetwork/WEB-INF/lib/: maven dependencies
with many excludes ...
- xml/schemas/
- WEB-INF/data/*.db
- WEB-INF/data/index/**
- ...
These appear to be a safety measure to avoid including files produced when src/main/webapp is used as a live directory by jetty below.

jetty

The jetty-maven-plugin defines web application using:

maven dependencies
target/geonetwork
src/main/webapp
../web-ui/src/main/resources/
target/webapp
jett-context.xml: used to prevent jetty scanning jars during startup

Observations:

This configuration has been setup to use the "live" src/main/webapp location and pick up any changes each time mvn process-resources is called above.
schemas plugin folders are included twice:
- in target/geonetwork via war definition
- in src/main/webapp/ via process-resource copy
jars are included twice:
- target/geonetwork libs folder
- as maven dependency

Schema Plugin Changes

Example pom.xml making use of maven-assembly-plugin and core-geonetwork project.version number from parent:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">

 <parent>
    <artifactId>schemas</artifactId>
    <groupId>org.geonetwork-opensource.schemas</groupId>
    <version>3.11.0-SNAPSHOT</version>
  </parent>
  <modelVersion>4.0.0</modelVersion>

  <artifactId>schema-iso19115-3.2018</artifactId>
  <name>GeoNetwork schema plugin for ISO19115-3:2018 standard</name>
  <description>
    ISO/TC 211 industry standard for geospatial metadata providing information about the identification, the extent, the quality, the spatial and temporal aspects, the content, the spatial reference, the portrayal, distribution, and other properties of digital geographic data and services.
  </description>

  <dependencies>
    <dependency>
      <groupId>org.geonetwork-opensource.schemas</groupId>
      <artifactId>schema-core</artifactId>
      <version>${project.version}</version>
    </dependency>
  </dependencies>

  <build>
    <resources>
      <resource>
        <directory>src/main/config/translations</directory>
        <targetPath>META-INF/catalog/locales</targetPath>
      </resource>
      <resource>
        <directory>src/main/resources</directory>
      </resource>
    </resources>
    <plugins>
      <plugin>
        <artifactId>maven-jar-plugin</artifactId>
        <executions>
          <execution>
            <id>test-jar</id>
            <goals>
              <goal>test-jar</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
      <plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <executions>
          <execution>
            <id>plugin-assembly</id>
            <phase>package</phase>
            <goals><goal>single</goal></goals>
            <inherited>false</inherited>
            <configuration>
             <appendAssemblyId>false</appendAssemblyId>
             <descriptors>
              <descriptor>src/assembly/schema-plugin.xml</descriptor>
             </descriptors>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Example schema-plugin.xml to package plugins folder:

<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.1.0 http://maven.apache.org/xsd/assembly-2.1.0.xsd">
  <id>plugin</id>
  <includeBaseDirectory>false</includeBaseDirectory>
  <formats>
    <format>zip</format>
  </formats>
  <fileSets>
    <fileSet>
      <directory>src/main/plugin/</directory>
      <outputDirectory></outputDirectory>
      <useDefaultExcludes>true</useDefaultExcludes>
    </fileSet>
  </fileSets>
</assembly>

See 4701 schemas/README.md for step by step instructions.

Build changes

As much as possible these build changes respect developers current workflow, using different maven plugins to accomplish the same goal (see web/README.md for details).

mvn jetty:run -Penv-dev

To pick up a change build your schema plugin using mvn install and update the running jetty application with:

mvn process-resources

A fallback option has been provided using the prior technique of copying files between folders:

mvn process-resources -DschemasCopy=true

To include additional schema plugins into schemas/pom.xml:

<profiles>
  <profile>
    <id>schema-iso19139.xyz</id>
    <activation>
    <file>
      <exists>iso19139.xyz</exists>
    </file>
    </activation>
    <modules>
      <module>iso19139.xyz</module>
    </modules>
  </profile>
</profiles>

To include additional schema plugin in web/pom.xml:

<profile>
   <id>schema-iso19115-xyz</id>
   <activation>
     <property><name>schemasCopy</name><value>!true</value></property>
     <file><exists>../schemas/iso19115-3.2018</exists></file>
   </activation>
   <dependencies>
     <dependency>
       <groupId>org.geonetwork-opensource.schemas</groupId>
       <artifactId>schema-iso19139.xyz</artifactId>
       <version>${project.version}</version>
     </dependency>
   </dependencies>
   <build>
     <plugins>
       <plugin>
         <groupId>org.apache.maven.plugins</groupId>
         <artifactId>maven-dependency-plugin</artifactId>
         <executions>
           <execution>
             <id>iso19115-xyz-resources</id>
             <phase>process-resources</phase>
             <goals><goal>unpack</goal></goals>
             <configuration>
               <encoding>UTF-8</encoding>
               <artifactItems>
                 <artifactItem>
                   <groupId>org.geonetwork-opensource.schemas</groupId>
                   <artifactId>schema-iso19115-xyz</artifactId>
                   <type>zip</type>
                   <overWrite>false</overWrite>
                   <outputDirectory>${schema-plugins.dir}</outputDirectory>
                 </artifactItem>
               </artifactItems>
               </artifactItems>
             </configuration>
           </execution>
         </executions>
       </plugin>
     </plugins>
   </build>
 </profile>

This approach automatically builds the associated plugin if the folder iso19139.xyz is present, use of add-schema.sh automates this change, see 4701 schemas/README.md.

As a result of this proposal there is no requirement to build metadata101 plugins as part of a local core-geonetwork build. Automatic activation is provided to preserve current developer workflow, if building a metadata101 module in isolation use of mvn jetty:run -Pschema-iso19115-xyz can be used to try it out.

Proposal Type:

Type: Improvement to existing feature
Module: schema plugins, metadata101, webapp, root

Voting History

This activity was covered by the Bolsena 2020 code sprint, see Bolsena 2020 Online slides for discussion and decisions.

Decision on version numbering:
- geonetwork: major/minor/revision format
- “3.10.4-SNAPSHOT”
Decision on schema-plugin version:
- Change: Use geonetwork version above
- Resolves transitive instability
Decision on schema-plugin packaging:
- publish both jar and zip to osgeo repository
Decision on metadata101/iso19115-3.2018
- Fold into core-geonetwork
- aside: Can be handled as separate pull request
Vote Proposed: TBA