Solved - Upgrade from 1.4 to 1.5 kills server

[Slydder]

How goes all?

I am admining a SEB server install for a school and I just attempted (for the second time) an upgrade to 1.5 from 1.4 and both times I can get the update but as soon as I tell docker to start the containers the server is no longer reachable (network is damaged). restarting the VM after that will start but without a working network stack.

Let me know how I can help you to figure out what is going wrong here.

thx,
chuck

[anhefti]

Hi chuck

We need more information to be able to figure out what is not working with the new Version on your Setup. Please can you tell us what setup do you use (dockerhub or old setup with Dockerfile) and how your setup is working. You can us present your docker-compose.yaml file, if you don't mind. And most valuable, the SEB Server log after the container has started up.

In the SEB Server logs after startup you should be able to see all relevant information about the startup, eventually database migrations and the SEB Server network connections and URL settings. If the SEB Server is starting up within the Docker container, it is most probably something wrong with the DB migration or network/URL settings. If the SEB Server is not starting up and the Docker container tries to start it up again and again, it would also be useful if you can get the logs or try to figure out what causes the Java process to stop starting up.

Best
Andreas

[Slydder]

Below is the install we used on a stock Debian Bullseye install.

cd /opt
mkdir sebserver
cd sebserver
git clone -b master https://github.com/SafeExamBrowser/seb-server-setup.git
cd seb-server-setup/docker/prod/bundled/dockerhub/
cat <<EOF > .env
SEBSERVER_PWD=
DB_SA_PWD=
EOF
docker-compose build --no-cache
docker-compose up -d
docker ps --all
docker logs $(docker ps --all|grep anhefti/seb-server|tr -s ' '| cut -d ' ' -f 1)

The upgrade path we are using is:

cd /opt/sebserver/seb-server-setup/docker/prod/bundled/dockerhub/
docker-compose down
cd /opt/sebserver/seb-server-setup
git fetch --tags
latestTag=$(git describe --tags ``git rev-list --tags --max-count=1``)
git stash
git checkout $latestTag
git stash apply
cd /opt/sebserver/seb-server-setup/docker/prod/bundled/dockerhub/
docker-compose build --no-cache
docker-compose pull
docker-compose up -d

At this point is where it all dies. Am having to wait to get a remote console to try and get access to the logs as this an offsite server.

I can verify taht $latestTag is actually 1.5.0

[anhefti]

Hi

If you have already a working setup with dockerhup, you do not need to fetch the seb-server-setup GIT repository anymore. This might not work since your stash and stash apply might scramble things up within the installation directory.
Instead you just can stop your working SEB Server setup with docker-compose down. Then open the docker-compose.yml file and adapt the docker image to the latest stable version as recommended in the installation documentation:

Then pull the new image with the update with docker-compose pull
And start-up again the SEB Server with docker-compose up -d

Then you should have a look to the startup logs if the DB migration was done properly when you update form one major version to another. And you should also check if your HTTP endpoint settings are correct.

What I miss in your setup is the HTTP endpoint settings as environment variables as it is mention in the installation guide:

https://seb-server-setup.readthedocs.io/en/latest/install-production.html#productive-setup-with-dockerhub point 3.
sebserver_webservice_http_external_scheme:
sebserver_webservice_http_external_servername:
sebserver_webservice_http_external_port:

Then there is already a new patch version 1.5.1. please also try to use that one or best, use 1.5-stable tag to always get the latest stable version of a major release.

Best
Andreas

[Slydder]

Andreas,

The external variables were set in the docker file and have been working fine since initial install

Also, changing 1.4-stable to 1.5-stable with a pull and up also kills the server. as soon as the up is issued I get "Creating network "seb-server-network" with the default driver" and then the server is no longer online.

we did a step by step standard install so not sure what is going on here. am trying to get a remote desk to I can get access to the VM window on the nutanix server.

[anhefti]

So if I understand this correctly, The whole server goes down while Docker tries to establish the internal Docker-Network: "seb-server-network"?

If so, this seems more like a Docker (installation) issue than an issue with the SEB Server Java server inside the Docker Container. Usually if something is wrong with the SEB Server Java App the seb-server container is there but will constantly try to restart the Java process within the container... or simple the Java process is running but with errors, but never cause issues on the Docker infrastructure.

If you are able to, please also try to analyze the docker installation and setup then or try to find out what causes your server to go down. Is there enough memory and disk-space left? Docker and Host-Server logs and so on.

[Slydder]

Here is the netstack prior to upgrade to 1.5

root@seb:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 50:6b:8d:c7:0e:1c brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 193.175.119.19/28 brd 193.175.119.31 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::526b:8dff:fec7:e1c/64 scope link
valid_lft forever preferred_lft forever
3: br-0ba6bc0b31a5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:49:6b:47:dd brd ff:ff:ff:ff:ff:ff
inet 172.23.0.1/16 brd 172.23.255.255 scope global br-0ba6bc0b31a5
valid_lft forever preferred_lft forever
inet6 fe80::42:49ff:fe6b:47dd/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:92:e7:e9:7e brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
6: vethff82af0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0ba6bc0b31a5 state UP group default
link/ether 76:17:23:42:91:a8 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::7417:23ff:fe42:91a8/64 scope link
valid_lft forever preferred_lft forever
8: veth727e2b6@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0ba6bc0b31a5 state UP group default
link/ether 96:1d:47:bc:34:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::941d:47ff:febc:3429/64 scope link
valid_lft forever preferred_lft forever
10: veth3263312@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0ba6bc0b31a5 state UP group default
link/ether ca:f2:55:5b:94:35 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::c8f2:55ff:fe5b:9435/64 scope link
valid_lft forever preferred_lft forever

After the update the stack looks like this:

root@seb:/opt/sebserver/seb-server-setup/docker/prod/bundled/dockerhub# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 50:6b:8d:c7:0e:1c brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 193.175.119.19/28 brd 193.175.119.31 scope global ens3
valid_lft forever preferred_lft forever
inet6 fe80::526b:8dff:fec7:e1c/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:92:e7:e9:7e brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
11: br-192927fcd7cb: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:68:4c:18:66 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-192927fcd7cb
valid_lft forever preferred_lft forever
inet6 fe80::42:68ff:fe4c:1866/64 scope link
valid_lft forever preferred_lft forever
13: vethd2daa8f@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-192927fcd7cb state UP group default
link/ether 6e:72:bf:b0:bd:e4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::6c72:bfff:feb0:bde4/64 scope link
valid_lft forever preferred_lft forever
15: veth9512f1e@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-192927fcd7cb state UP group default
link/ether 82:8a:ee:60:29:92 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::808a:eeff:fe60:2992/64 scope link
valid_lft forever preferred_lft forever
17: veth21ac8fa@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-192927fcd7cb state UP group default
link/ether 5e:96:22:eb:ff:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::5c96:22ff:feeb:ffd6/64 scope link
valid_lft forever preferred_lft forever

As you can see, the bridge interface is now in a different subnet which conflicts with our VPN subnet. So the update does complete successfully but it is no longer usable because the netstack is broken.

Basically we need to get rid of the IP 172.18.0.1 and go back to 172.23.0.1 which should get connection back to the network.

[anhefti]

Thanks for the update.

I'm not very familiar with networking and the operational setup of Docker and SEB Server is not part of the SEB Server development project. For my knowledge there has nothing been changed in the docker-compose network setup within the docker-compose file of SEB Server since version 1.4 and dockers network bridge configuration is not part of the SEB Server installation but part of the Docker installation:

You can additionally set a driver for the network within the docker-compose file like described here: https://docs.docker.com/compose/networking/ but I don't know if this is solving your problem.

The IP Addresses you mentioned above are not part of the SEB Server setup configuration but managed by Docker and the OS. Maybe something has changed there. Unfortunately I cannot really help here since this seems to be out of scope and due to your specific docker and host setup.

[Slydder]

Here is the killshot.

root@seb:/etc/docker# docker network inspect seb-server-network
[
{
"Name": "seb-server-network",
"Id": "192927fcd7cb307907bcf9e13828c7a08c2f4fd1f6364bc499e67130d35d0b09",
"Created": "2023-08-14T13:31:38.349556818+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": true,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"100c51b55b72328ae2be55a1b94893da8bb076f90f84eab5d421969c4cc6fd09": {
"Name": "seb-server-proxy",
"EndpointID": "ce891abe5073563844ae1d58b251b58de9611c07610c749f97546621933fe652",
"MacAddress": "02:42:ac:12:00:04",
"IPv4Address": "172.18.0.4/16",
"IPv6Address": ""
},
"a43df8611b22dd7689087acf0ccf1b23b866750746f754c9b2df4317f594af48": {
"Name": "seb-server-mariadb",
"EndpointID": "e6e3bb880ad6a6fa9306ab75157a49bc1c646a3085a6ce5716ff9896fd2fd20d",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
},
"ea512ba92e7066ead07620995d30c3df8ceb3be3c15232929ae4014717aa2a43": {
"Name": "seb-server",
"EndpointID": "58e5c1146c792879d159f4413416b90b8fa231ab984a20a7177b0330fc2ab6b4",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {},
"Labels": {
"com.docker.compose.network": "seb-server-network",
"com.docker.compose.project": "dockerhub",
"com.docker.compose.version": "1.25.0"
}
}
]

[Slydder]

OK. a small adjustment to docker-compose.yml will get the network back where it was prior to the upgrade. This will only be needed if the upgrade breaks networking/routing in the installation.

networks:
seb-server-network:
name: seb-server-network
ipam:
config:
- subnet: 172.23.0.0/24