[Bug]: cs2server. Monitor will fail when server is actually on. Then it will send quit and restart server.

[tk1114632]

User story

Need help to fix.

Game

cs2server

Linux distro

Ubuntu 22.04

Command

command: monitor

Further information

Monitor will fail when server is actually on. Then it will send quit and restart server.

Log shows: "PIDS with old type tmux session are running." What is it meaning?

Relevant log output

Script log:

May 02 20:33:19.085 cs2server-2: MONITOR: INFO: Checking session: CHECKING
May 02 20:33:19.297 cs2server-2: MONITOR: ERROR: Checking session: ERROR
May 02 20:33:19.298 cs2server-2: MONITOR: ERROR: Checking session: PIDS with old type tmux session are running
May 02 20:33:19.298 cs2server-2: MONITOR: ERROR: Checking session: Killing session with the session name cs2server-2
May 02 20:33:20.971 cs2server-2: STOP: INFO: Graceful: sending "quit"
May 02 20:33:25.678 cs2server-2: STOP: PASS: Graceful: sending "quit": OK: 3 seconds
May 02 20:33:26.182 cs2server-2: STOP: INFO: tmux kill-session: cs2server-2: Flaming Rose
May 02 20:33:27.289 cs2server-2: STOP: PASS: Stopped Flaming Rose
May 02 20:33:28.327 cs2server-2: START: INFO: Using anonymous Steam login
May 02 20:33:29.036 cs2server-2: START: INFO: Rotating log files

Steps to reproduce

1. crontab: * * * * * ~/cs2server-X m
2. After server runs for random period, server is restarted by monitor.

[vg2808]

I also came across such a mistake.. Any fix?

[mayersgamer]

me too , server restart with same error when using 2 instances with the same installation
i have set monitor to verify every 2 minutes so i think i will remove it because its causing the restart.

Nov 08 17:34:07.865 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:34:07.869 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:34:07.871 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:36:02.697 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:36:03.592 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:36:05.148 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:36:05.361 cs2server: MONITOR: PASS: Checking session: OK
Nov 08 17:36:05.363 cs2server: MONITOR: INFO: gamedig is not installed
Nov 08 17:36:05.364 cs2server: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 08 17:36:06.868 cs2server: MONITOR: INFO: Querying port: gsquery: 31.56.0.68:27015 : 1 : QUERYING
Nov 08 17:36:07.092 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:36:07.097 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:36:07.099 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:38:02.009 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:38:03.204 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:38:04.757 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:38:04.971 cs2server: MONITOR: PASS: Checking session: OK
Nov 08 17:38:04.974 cs2server: MONITOR: INFO: gamedig is not installed
Nov 08 17:38:04.976 cs2server: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 08 17:38:06.481 cs2server: MONITOR: INFO: Querying port: gsquery: 31.56.0.68:27015 : 1 : QUERYING
Nov 08 17:38:06.705 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:38:06.711 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:38:06.712 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:40:02.818 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:40:03.542 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:40:05.094 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:40:05.309 cs2server: MONITOR: PASS: Checking session: OK
Nov 08 17:40:05.311 cs2server: MONITOR: INFO: gamedig is not installed
Nov 08 17:40:05.313 cs2server: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 08 17:40:06.816 cs2server: MONITOR: INFO: Querying port: gsquery: 31.56.0.68:27015 : 1 : QUERYING
Nov 08 17:40:07.042 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:40:07.048 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:40:07.049 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:42:02.083 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:42:02.790 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:42:04.339 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:42:04.552 cs2server: MONITOR: PASS: Checking session: OK
Nov 08 17:42:04.554 cs2server: MONITOR: INFO: gamedig is not installed
Nov 08 17:42:04.555 cs2server: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 08 17:42:06.059 cs2server: MONITOR: INFO: Querying port: gsquery: 31.56.0.68:27015 : 1 : QUERYING
Nov 08 17:42:06.281 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:42:06.286 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:42:06.287 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:44:02.767 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:44:03.501 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:44:05.052 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:44:05.266 cs2server: MONITOR: PASS: Checking session: OK
Nov 08 17:44:05.268 cs2server: MONITOR: INFO: gamedig is not installed
Nov 08 17:44:05.270 cs2server: MONITOR: INFO: https://docs.linuxgsm.com/requirements/gamedig
Nov 08 17:44:06.773 cs2server: MONITOR: INFO: Querying port: gsquery: 31.56.0.68:27015 : 1 : QUERYING
Nov 08 17:44:06.999 cs2server: MONITOR: PASS: Querying port: gsquery: 31.56.0.68:27015 : 1 : OK
Nov 08 17:44:07.003 cs2server: MONITOR: INFO: LinuxGSM version: v24.3.2
Nov 08 17:44:07.005 cs2server: MONITOR: PASS: core_exit.sh exiting with code: 0
Nov 08 17:46:01.875 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:46:02.541 cs2server: MONITOR: PASS: Using cached IP as public IP address
Nov 08 17:46:04.095 cs2server: MONITOR: INFO: Checking session: CHECKING
Nov 08 17:46:04.309 cs2server: MONITOR: ERROR: Checking session: ERROR
Nov 08 17:46:04.310 cs2server: MONITOR: ERROR: Checking session: PIDS with old type tmux session are running
Nov 08 17:46:04.312 cs2server: MONITOR: ERROR: Checking session: Killing session with the session name cs2server
Nov 08 17:46:04.389 cs2server: RESTART: PASS: Using cached IP as public IP address
Nov 08 17:46:04.998 cs2server: STOP: PASS: Using cached IP as public IP address
Nov 08 17:46:05.567 cs2server: STOP: PASS: Using cached IP as public IP address
Nov 08 17:46:06.107 cs2server: STOP: INFO: Graceful: sending "quit"
Nov 08 17:46:10.819 cs2server: STOP: PASS: Graceful: sending "quit
Nov 08 17:46:11.820 cs2server: START: PASS: Using cached IP as public IP address
Nov 08 17:46:11.869 cs2server: START: INFO: Using anonymous Steam login
Nov 08 17:46:12.066 cs2server: START: PASS: Using cached IP as public IP address
Nov 08 17:46:12.614 cs2server: START: INFO: Rotating log files

crontab :
*/2 * * * * /home/server1/cs2server monitor > /dev/null 2>&1
*/30 * * * * /home/server1/cs2server update > /dev/null 2>&1
0 0 * * 0 /home/server1/cs2server update-lgsm > /dev/null 2>&1
@reboot /home/server1/cs2server start > /dev/null 2>&1
0 6 * * * /home/server1/cs2server restart > /dev/null 2>&1
*/2 * * * * /home/server1/cs2server-2 monitor > /dev/null 2>&1
*/30 * * * * /home/server1/cs2server-2 update > /dev/null 2>&1
@reboot /home/server1/cs2server-2 start > /dev/null 2>&1
0 6 * * * /home/server1/cs2server-2 restart > /dev/null 2>&1

ok maybe it was happning before when i had 2 servers with 2 diffrent users and diffrent instalation i used to see steamgamemanagment token key being refreshed 2-3 times a day and it shoudnt because i set the servers to restart only at 6 am every day .

so i think monitor failled to monitor and leads the servers to quit and restart .

im going to try for monitor once every hour and see if steam game manager token will refresh the token of my servers outside the sheduled time for restart.

[MrLenin]

I have run into this issue as well monitoring multiple factorio server instances from a single install. With a single instance everything works fine indefinitely, but as soon as you monitor a second instance, it will fail and restart anywhere between 5 minutes and 2 hours over and over again.

I found it interesting this seemed to only affect the first server, the second server never experienced the issue even once over weeks.

I added some additional logging to the monitor script to see what the expanded pgrep command is and the value of its output and they are exactly what you would expect for this error to occur.

I am reasonably sure at this point that the issue almost exclusively affects unnumbered instances in a multi-instance setup where the pgrep expression for the unnumbered instance can match on the command lines for the pgrep's being run for the monitoring of additional instances.

I am testing a modified version of the monitoring module where the problematic pgrep expression has been modified to add a '^' to the beginning to ensure it is explicitly matching the start of command line, I suspect this will resolve the problem.

It may be a good idea to repeat this for other applicable pgreps in the codebase if it is found to correct this problem.

[MrLenin]

It hasn't quite been two hours yet since the last restart (4:20:17 EST), but I am feeling reasonably confident at this point my change resolves the underlying issue.

It should be noted perhaps that my suggested fix might not work in the hopefully uncommon situation where the monitor module is able to see instances across multiple users.

[h3o66]

crontab :
*/2 * * * * /home/server1/cs2server monitor > /dev/null 2>&1
*/30 * * * * /home/server1/cs2server update > /dev/null 2>&1
0 0 * * 0 /home/server1/cs2server update-lgsm > /dev/null 2>&1
@reboot /home/server1/cs2server start > /dev/null 2>&1
0 6 * * * /home/server1/cs2server restart > /dev/null 2>&1
*/2 * * * * /home/server1/cs2server-2 monitor > /dev/null 2>&1
*/30 * * * * /home/server1/cs2server-2 update > /dev/null 2>&1
@reboot /home/server1/cs2server-2 start > /dev/null 2>&1
0 6 * * * /home/server1/cs2server-2 restart > /dev/null 2>&1

Witch crontab do you mean with this one ?
The user crontab via crontab -e or maybe /etc/crontab ?

[MrLenin]

I have only used the user crontab.
Change line 204 of ~/lgsm/modules/command_monitor.sh from
elif [ "$(pgrep -fc -u "${USER}" "tmux new-session -d -x ${sessionwidth} -y ${sessionheight} -s ${sessionname}")" != "0" ]; then to
elif [ "$(pgrep -fc -u "${USER}" "^tmux new-session -d -x ${sessionwidth} -y ${sessionheight} -s ${sessionname}")" != "0" ]; then
this should resolve this and only this specific problem, but it could potentially reoccur with any other pgrep expression that isn't explicitly matching against that beginning of the command line

Over three hours without a restart now.

[MrLenin]

Thinking about this some more, while my change does fix this particular issue, the code is still ultimately bugged as it would still trigger on the unnumbered instance for any numbered instance with a tmux launched using the old parameters, so it doesn't work as intended even.