Hermes the Warehouse & Logistics Service

Hermes WLS (Warehouse and Logistics Service) functions as a middleware between NLNs (National Library of Norway) catalogues and storage systems. The goal with the service is to unite all the storage systems and catalogues used at NLN with a common interface.

Benefits of this approach are:

• Decoupling of the storage systems and catalogues, which make it easier to change systems if needed.
• Makes it easier for end users to access material stored in different systems.
• Storage systems don't need to know which catalogue to inform about changes, as the service will handle this.

More features and benefits will be added as the service is developed.

Table of Contents

1. Hermes the Warehouse & Logistics Service
2. Technologies
3. Running the Application
   1. Building and Running Locally
      1. Using Maven
      2. Using Docker
      3. Using an IDE
   2. Running Tests
      1. Running Tests in the Pipeline
      2. Running Tests in an IDE
4. Usage
5. Dependencies
   1. Local Dependencies
   2. Deployment Dependencies
6. Development
7. Configuration
8. Deployment
   1. Deploying to Staging Environment
   2. Deploying to Production Environment
9. Contact
10. License

Technologies

Hermes WLS uses the following technologies:

• Eclipse Temurin for the Java runtime.
• Kotlin for the application code.
• Maven for project management.
• Spring Boot for the application framework.
• MongoDB for data storage.
• Keycloak for client authentication and authorization.
• Swagger for API documentation.
• Docker for containerization.
• Harbor for container registry.
• Kubernetes for deployment and orchestration.
• Vault for secrets management.
• GitHub Actions for CI/CD.

As of now the service is in the early stages of development, and is not yet in production. Therefore the technologies listed above are subject to change. Check the pom.xml file for the most up-to-date list of dependencies. As well as the Dockerfile for the current Docker image setup. And lastly the Kubernetes deployment for the current deployment setup. You might also want to check the GitHub Actions for the current CI/CD setup.

Running the Application

The Warehouse Logistics Service is a Spring Boot application that can be run locally or in a container. It is recommended to use Docker for local testing, as the service is designed to run in a containerized environment. Additionally, this service depends on other applications, such as MongoDB, which can be spun up using provided Docker Compose file. For development an IDE such as IntelliJ IDEA is recommended.

Building and Running Locally

Using Maven

Use following commands to build and run the application locally:

# Package the application, will execute the tests too
mvn clean package

# Run it locally
java -jar target/wls.jar

Using Docker

After building the JAR file, it can be used to build a Docker image using the following commands:

# Move the jar to the Docker directory
cp target/wls.jar docker/wls.jar

# Use Docker Buildx to build the Docker Image
docker buildx build --platform linux/amd64 -t wls:latest docker/

Caveats:

• When building the Docker image outside of NLNs network the build will fail, as it won't be able to access internal Harbor instance. In this case change the FROM line in the Dockerfile to FROM eclipse-temurin:21-jdk-alpine and build the image locally.
• If you are outside of NLNs network your tests will fail, as an image for the dummy SynQ server is required for tests. There is currently no replacement available for this, except for building the image for Dummy SynQ manually and pointing to it in places where it's used.
• Do not attempt to push the image to Harbor manually, as it will fail. The image is built and pushed to Harbor automatically by the CI/CD pipeline.

A pre-built image can be found on NLNs internal Harbor instance under the mlt namespace. The images are built based on the main branch as well as project tags, and can be pulled using the following command:

# Pull the latest image
docker pull harbor.nb.no/mlt/wls:latest

# Or pull a specific tag (either a GitHub tag or "main" for the latest main branch image)
docker pull harbor.nb.no/mlt/wls:<TAG>

With the image either built or pulled, it can be run using the following command:

docker run -p 8080:8080 -e SPRING_PROFILES_ACTIVE="local-dev" harbor.nb.no/mlt/wls:<TAG>

Using an IDE

For local development and testing an IDE like IntelliJ IDEA is recommended. Its default Spring run configuration for the application works well. To test the service with authentication make sure that dev version of the Keycloak is running, and set the SPRING_PROFILES_ACTIVE variable to local-dev. In case you can't/won't use local Keycloak instance, then provide the KEYCLOAK_ISSUER_URI variable (see below) and set the SPRING_PROFILES_ACTIVE variable to stage. Keycloak in dev and stage is set up with a test client wls.

Running Tests

In order to run the tests, use the following command:

mvn clean test

It should run all the tests in the project and provide a report at the end. You can see the results in both the console and in the target/surefire-reports directory.

Running Tests in the Pipeline

The CI/CD pipeline will run the tests automatically when a pull request is created. It will create a report and provide it in the pull request. It can be accessed by clicking on the Details link in the Checks section for the Deploy Project / JUnit Tests check of the pull request.

Running Tests in an IDE

In an IDE like IntelliJ IDEA, the tests can be run by right-clicking on the src/test/kotlin directory in the project and selecting Run tests in 'kotlin'. Further configuration can be done in the Run/Debug Configurations menu.

In order to run tests with authentication, you will need to set the spring.profiles.active system property to local-dev in the run configuration. This can be easily done by adding the following to the VM options field in the run configuration:

-ea  -Dspring.profiles.active=local-dev

Of course this requires you to have the dependent services running locally, see the Local Dependencies section for more information. In short you will need to start the services using the provided Docker Compose file.

Usage

Hermes WLS provides a REST API for interacting with the service. The API is documented using Swagger, and can be accessed by running the application and navigating to the following URL: http://localhost:8080/swagger

As the staging and production environments are deployed on internal networks, the deployed API is not accessible from the outside. If you need to access the API in these environments, you will need to use a VPN connection to NLN's network. The API is accessible at the usual URL, with the /hermes suffix.

Dependencies

Local Dependencies

Regardless of what method you used to run the Hermes WLS, it has other services and applications that it depends on. In order to run these, use the provided Docker Compose file. This will spin up the following services:

• MongoDB: database for the application
  • Can be accessed through provided mongo-express service at: http://localhost:8081
  • Use the following credentials to log in:
    • Username: root
    • Password: toor
• Keycloak: authentication and authorization service for the application
  • Can be accessed at: http://localhost:8082
  • Use the following credentials to log in:
    • Username: root
    • Password: toor

To start the services, run the following command:

docker compose -f docker/compose.yaml -p wls-local up -d

And to stop the services, run the following command:

docker compose -p wls-local down

Deployment Dependencies

In addition to the local dependencies, the Hermes WLS also depends on the following services:

• Kubernetes: for deployment and orchestration of the application
• Harbor: for hosting the Docker image of the application
• Vault: for secrets management in the deployment pipeline

All of these services are managed by the NLN's Platform team, and are not needed for local development. However, they are needed for deployment of the application to the staging and production environments. MongoDB and Keycloak are also maintained by the Platform team, and are used in the deployed application.

Development

The development of the Hermes WLS is done in "feature branches" that are merged into the main branch. Name each feature branch after its JIRA code followed by a short summary of the feature. For example mlt-0018-add-readme.

Make sure that your development tool supports the EditorConfig standard, and use the included .editorconfig file. IntelliJ IDEA supports formatting the code according to the .editorconfig file, and can be set up in the Editor settings.

Additionally you should run the tests, and run Spotless to ensure that the code is formatted correctly. To run tests and spotless, use the following commands:

mvn clean test
mvn spotless:apply

The CI/CD pipelnine will run these commands automatically when a pull request is created. Although it's better to run them on your machine before pushing as it will save time and resources.

Configuration

The following environment variables are relevant to configuring the application:

• KEYCLOAK_ISSUER_URI: Is used to point at the Keycloak server used for authentication (required)
• KEYCLOAK_TOKEN_AUD: Is used to set the audience of the Keycloak JWT token, it must match with the issued token audience value, which is different between environments (required)
• SPRING_PROFILES_ACTIVE: Is used to set the active Spring profile, use local-dev or stage for testing authentication (optional, default is pipeline)
• MONGODB_USERNAME: Is the username used for connecting to MongoDB (required)
• MONGODB_PASSWORD: Is the password used for connecting to MongoDB (required)
• MONGODB_DATABASE: Is the name of the database that Hermes should use (required)
• MONGODB_URL: Is the base URL at which a MongoDB instance runs (required)
• SYNQ_BASE_URL: Is the base URL used for communicating against SynQ (required)

Deployment

The section Running the Application describes how to run the application locally. Therefore this section will focus on how to deploy the application to the staging and production environments.

Deployment to both environments is handled by their respective Kubernetes deployment files. They describe the deployment, service, and ingress for the application. There is very little difference between the two files, as the environments are very similar. They mostly deal with resource limits and requests, as well as the number of replicas.

In both cases the deployment is handled by the CI/CD pipeline.

Deploying to Staging Environment

To deploy the application to the staging environment, push new changes to the main branch. The repository is set up to only accept merges to the main branch through pull requests. Therefore in order to deploy to the staging environment, create a pull request and merge it to the main branch. Actions in pull request should test the application to make ensure that it is working as expected.

When a pull request is merged to the main branch, the CI/CD pipeline will build the application, create a Docker image, and push it to Harbor. Then the image will be deployed to the staging cluster in NLN's internal Kubernetes system.

Deploying to Production Environment

To deploy the application to the production environment, create a new tag in the repository. The tag should be in the format vX.Y.Z, where X, Y, and Z are numbers, following the semantic versioning standard.

This will trigger the CI/CD pipeline which will deploy the application to the production environment. This is quite similar to the staging deployment.

Contact

The project is maintained by the National Library of Norway organization. It is owned by the "Warehouse and Logistics" team (MLT).

For questions or issues, please create a new issue in the repository.

You can also contact the team by email at mlt at nb dot no.

License

This project is licensed under the MIT License.