Align to GeoNode master (4.2.0) and improve/fix docker initialization #457

afabiani · 2023-07-27T09:00:06Z

This PR introduces improvements and fixes to docker-compose and initialization scripts in general:

Removing some unused variables.
Removing some unused and obsolete containers, such as Jenkins.
Removing the incorrectly written and non-functioning part for setting the password of GeoServer.
Inverting the startup dependency from GeoServer -> Django to Django -> GeoServer.
Ensuring that OAUTH2 is properly initialized on restart in all its parts.
Enhancing NGINX to make sure that GeoServer functions well behind any proxy, including HTTPS.

Some implementation details:

Environment file

DOCKERHOST and DOCKER_HOST_IP variables removed as useless and misleading. Instead, it exposed NGINX_BASE_URL which was calculated automatically and badly.

The variables

GEONODE_LB_HOST_IP=django
GEONODE_LB_PORT=8000
GEOSERVER_LB_HOST_IP=geoserver
GEOSERVER_LB_PORT=8080

are important, they are used by NGINX to create the upstream proxyPass and by GeoServer to create the internal OAuth2 endpoints. They represent the name and port of the GeoNode and GeoServer containers. In general it is not necessary to change them, but in an environment like Microsoft Azure these must be set to the address of the subnet as the names are not internally resolved.

Increased MAX_DOCUMENT_SIZE=200 because the previous default of 2Mb did not allow rendering even one document.

Updated create-envfile.py to also enable staging mode to test HTTPS on localhost, which was previously totally ignored.

Dockerfiles

Removed useless stuff and started it from geonode/geonode-base:latest-ubuntu-22.10

Readme

Added a Quick Start section with docker

Docker Compose

Fixed healthchecks which are now faster and more efficient and switched Django -> GeoServer startup dependency, because it's the Django entrypoint that sets OAutg2 parameters in the GEOSERVER_DATA_DIR before it starts.

Updated to POSTgresql 15

UWSGI + NGINX

Enabled proxyPass instead of uwsgiPass because we set headers correctly in geonode.conf only with proxyPass

GeoServer docker entry points

Fixed the .sh to set OAuth2 which previously worked badly and only partially checking when there was the localhost value. Now they update everything and always.

nginx/geonode.conf.envsubst

Made proxy passes dynamic

set $upstream $GEOSERVER_LB_HOST_IP:$GEOSERVER_LB_PORT;

set $upstream $GEONODE_LB_HOST_IP:$GEONODE_LB_PORT;

Updated the GeoServer conf so that the GUI works fine behind any proxy.

A general reformat to the black code
Version bump

from 4.0.0.final to 4.2.0.dev

src/tasks.py

    try:
-        r = requests.request('get', url, verify=False)
+        r = requests.request("get", url, verify=False)


src/project_name/__init__.py



 def get_version():
-    import {{project_name}}.version
-    return {{project_name}}.version.get_version(__version__)
+    import {{ project_name }}.version


…bles (starting the process at least)

ridoo · 2023-07-31T09:42:46Z

@afabiani these are quite good changes! However, the PR raises a maintenance question:

What do you actually recommend when users have to decide between a GeoNode Fork or the creation of a separate Geonode project via geonode-project template?

The main problem I see is to keep track of all the changes you make on this repository (e.g. those within this PR). Once a project has been created, it gets detached from version control. If a user decides to upgrade their GeoNode project, they would have to follow the commit history and apply all changes manually, right? Wouldn't it be better to just use a GeoNode fork so the changes become visible once you do a merge/pull?

Is there is a discussion (issue/documentation) about Forking vs. GeoNode-Project somewhere? Some good practice guidance would be helpful too.

I'd be happy for any pointer :-)

afabiani · 2023-07-31T09:45:28Z

Those changes won't ba backported. Those will be eventually part of the 4.2.0 train, currently under development.

ridoo · 2023-07-31T09:47:44Z

Ok .. but what about mainaining a project which got generated from the geonode-project repository? When upgrading to 4.2.0 .. I would have to go through the commit history and apply the changes manually on that project. Or do I miss something?

afabiani · 2023-07-31T09:52:59Z

The idea would be to update the documentation of the 4.2.0 release with a specific section for this topic. However we can open a discussion on the mailing list to further investigate the problem and hear from the community if there are some suitable proposals/ideas.

ridoo · 2023-07-31T10:00:19Z

However we can open a discussion on the mailing list to further investigate the problem and hear from the community if there are some suitable proposals/ideas.

You mean [email protected]? I am subscribed, but it seems it makes not much noise in my inbox. However, I will open a discussion thread there.

edit: mail address .. geonode-devel is the right one -- in fact I only found the geonode-devel-requests address in my inbox

ridoo · 2023-07-31T10:51:32Z

However, I will open a discussion thread there.

@afabiani my email awaits approval 🤷‍♂️

ridoo · 2023-08-22T13:14:35Z

docker/geoserver/tasks.py

+#
+#########################################################################
+import os
+import os


Remove redundant import

…o use the correct variable for the "geofenceDataSource.username"

ridoo · 2023-08-22T09:37:45Z

docker/geoserver/tasks.py

+    GEOSERVER_ADMIN_USER = os.getenv("GEOSERVER_ADMIN_USER", "admin")
+    GEOSERVER_ADMIN_PASSWORD = os.getenv("GEOSERVER_ADMIN_PASSWORD", "geoserver")
+    GEOSERVER_FACTORY_PASSWORD = os.getenv("GEOSERVER_FACTORY_PASSWORD", "geoserver")
+    geoserver_rest_baseurl = f"http://localhost:{GEOSERVER_LB_PORT}/geoserver/rest"


a strayed localhost, I suspect?!

Nope it's ok. It will always refer to the local container.

giohappy · 2023-08-22T11:22:48Z

@ridoo if you agree I would merge the PR

ridoo · 2023-08-22T11:36:06Z

@giohappy if it is ok for you, I'd take a deeper look later this day, before just saying "ok".

github-advanced-security · 2023-08-22T13:07:52Z

This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on this overview. Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out the documentation.

ridoo

I finished review .. however, I touched some parts which relate to Geonode core as well (the PR puts the changes made upstream here) .

However, I did not want to block the PR.

.env.sample

Dockerfile

ridoo · 2023-08-22T12:36:02Z

docker-purge.sh

+docker kill $(docker ps -q); docker rm $(docker ps -a -q); docker rmi $(docker images -q); docker volume ls -qf dangling=true | xargs -r docker volume rm; docker system prune -a
+docker volume prune


This file is quite dangerous as it removes everything non-geonode related, too! To my experience first-time users do not understand the implications immediately. Having a cleaned Docker setup is afterwards, well, .. surprising, at least.

Leverage the --filter flag:

# delete all containers with name including "thuenen" docker rm $(docker ps -aq --filter=name=thuenen) # delete all volumes with name including "thuenen" docker volume rm $(docker volume ls -q --filter name=thuenen) # delete all images having a `4.0` tag starting with `geonode` docker rmi $(docker images -q --filter=reference='geonode/*:4.0')

docker/geoserver/entrypoint.sh

ridoo · 2023-08-22T13:14:35Z

docker/geoserver/tasks.py

+#
+#########################################################################
+import os
+import os


Remove redundant import

docker/nginx/Dockerfile

src/uwsgi.ini

- Align to GeoNode master (4.2.0)

d341641

github-advanced-security bot found potential problems Jul 27, 2023

View reviewed changes

src/tasks.py

try:

r = requests.request('get', url, verify=False)

r = requests.request("get", url, verify=False)

Check failure

Code scanning / CodeQL

Request without certificate validation

This request may run without certificate validation because [it is disabled](1).

afabiani added 5 commits July 27, 2023 11:21

- Align to GeoNode master (4.2.0)

9b42323

- Align to GeoNode master (4.2.0)

eb012d8

- Align to GeoNode master (4.2.0)

1e4d39a

- Typo

4b0eb92

- Code formatting

5f00a32

github-advanced-security bot found potential problems Jul 28, 2023

View reviewed changes

src/project_name/__init__.py

def get_version():

import {{project_name}}.version

return {{project_name}}.version.get_version(__version__)

import {{ project_name }}.version

Check failure

Code scanning / CodeQL

Syntax error

Syntax Error (in Python 3).

afabiani added 2 commits July 29, 2023 00:09

- Optimizing and fixing docker stack; removing unneeded/unused varia…

e6be49d

…bles (starting the process at least)

Update __init__.py

cba3e57

afabiani requested review from mattiagiupponi and giohappy July 29, 2023 08:18

afabiani self-assigned this Jul 29, 2023

afabiani added this to the 4.2.0 milestone Jul 29, 2023

afabiani changed the title ~~- Align to GeoNode master (4.2.0)~~ Align to GeoNode master (4.2.0) and improve/fix docker initialization Jul 29, 2023

afabiani added 2 commits July 31, 2023 12:05

Typo on GEOSERVER_JAVA_OPTS

813c936

Update docker-compose-test django healthcheck with retries

28ab6e2

giohappy added 7 commits August 21, 2023 10:00

drop PUBLIC_PORT and other unused Dockerfile env vars

9e58413

disable abbreviated params

f6185a8

Removed legacy code

c0c4754

ported fixes to monitoring fixtures

7494654

Implement password configuration at init time

39ccc52

Removed Azure vars from .env sample

de4d1ac

Add issing headers

04943eb

github-advanced-security bot found potential problems Aug 21, 2023

View reviewed changes

afabiani added 2 commits August 21, 2023 19:14

- Fix: the oauth2 "logoutUri" should match the public url too

a32750e

- Fix: update the "geofence-datasource-ovr.properties.j2" in order t…

a0c5b4b

…o use the correct variable for the "geofenceDataSource.username"

ridoo reviewed Aug 22, 2023

View reviewed changes

set nginx image version

89bc888

giohappy requested a review from ridoo August 22, 2023 11:19

giohappy force-pushed the align_with_geonode_master branch from e37731a to 89bc888 Compare August 22, 2023 11:25

giohappy approved these changes Aug 22, 2023

View reviewed changes

giohappy force-pushed the align_with_geonode_master branch from fdee9e6 to 89bc888 Compare August 22, 2023 11:44

giohappy force-pushed the align_with_geonode_master branch from 613dfbc to 89bc888 Compare August 22, 2023 13:09

Fixes CodeQL

8c8f215

giohappy self-requested a review August 22, 2023 13:11

giohappy approved these changes Aug 22, 2023

View reviewed changes

ridoo reviewed Aug 22, 2023

View reviewed changes

giohappy added 6 commits August 22, 2023 16:05

Improvements to create-envfile documentation

1034125

Cumulative improvements

a5a780a

Give humane names to Geoserver variables

3393da2

Use GEOSERVER_PUBLIC_LOCATION

1a572c4

Remove geoserver_ui variable from .env.sample

634cb03

moved codeql-config to its folder to avoid trigger false workflow

55603ce

giohappy merged commit c4430ba into master Aug 22, 2023
4 checks passed

giohappy deleted the align_with_geonode_master branch August 22, 2023 17:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Align to GeoNode master (4.2.0) and improve/fix docker initialization #457

Align to GeoNode master (4.2.0) and improve/fix docker initialization #457

afabiani commented Jul 27, 2023 •

edited

Loading

ridoo commented Jul 31, 2023

afabiani commented Jul 31, 2023

ridoo commented Jul 31, 2023

afabiani commented Jul 31, 2023

ridoo commented Jul 31, 2023 •

edited

Loading

ridoo commented Jul 31, 2023

ridoo Aug 22, 2023

ridoo Aug 22, 2023

afabiani Aug 22, 2023

giohappy commented Aug 22, 2023

ridoo commented Aug 22, 2023

github-advanced-security bot commented Aug 22, 2023

ridoo left a comment •

edited

Loading

ridoo Aug 22, 2023

ridoo Aug 22, 2023

		docker kill $(docker ps -q); docker rm $(docker ps -a -q); docker rmi $(docker images -q); docker volume ls -qf dangling=true \| xargs -r docker volume rm; docker system prune -a
		docker volume prune

Align to GeoNode master (4.2.0) and improve/fix docker initialization #457

Align to GeoNode master (4.2.0) and improve/fix docker initialization #457

Conversation

afabiani commented Jul 27, 2023 • edited Loading

ridoo commented Jul 31, 2023

afabiani commented Jul 31, 2023

ridoo commented Jul 31, 2023

afabiani commented Jul 31, 2023

ridoo commented Jul 31, 2023 • edited Loading

ridoo commented Jul 31, 2023

ridoo Aug 22, 2023

Choose a reason for hiding this comment

ridoo Aug 22, 2023

Choose a reason for hiding this comment

afabiani Aug 22, 2023

Choose a reason for hiding this comment

giohappy commented Aug 22, 2023

ridoo commented Aug 22, 2023

github-advanced-security bot commented Aug 22, 2023

ridoo left a comment • edited Loading

Choose a reason for hiding this comment

ridoo Aug 22, 2023

Choose a reason for hiding this comment

ridoo Aug 22, 2023

Choose a reason for hiding this comment

afabiani commented Jul 27, 2023 •

edited

Loading

ridoo commented Jul 31, 2023 •

edited

Loading

ridoo left a comment •

edited

Loading