Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Worker does not exit on EHOSTUNREACH error? #62

Open
mStirner opened this issue Oct 28, 2024 · 2 comments
Open

Worker does not exit on EHOSTUNREACH error? #62

mStirner opened this issue Oct 28, 2024 · 2 comments
Labels
bug Something isn't working investigate

Comments

@mStirner
Copy link
Member

Trying to connect to a invalid/non exisiting/offline tcp socket/server, causes to spwan worker, but do not exit.
This creates a memory leak.

Related/debugging information: #61
A watch on watch -n 10 "systemctl status connector" and trying to connect to a offline raspberry increases the Tasks number continusly.

● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 87 (limit: 11107)
     Memory: 645.1M
        CPU: 17min 31.599s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:47:48 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:47:51 open-haus.lan node[51700]: [2024.10.28 - 16:47.51.943][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:47:51 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:47:51 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:47:51 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:47:51 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:47:51 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:47:51 open-haus.lan node[51700]:   port: 8080
Oct 28 16:47:51 open-haus.lan node[51700]: }
Oct 28 16:47:51 open-haus.lan node[51700]: [2024.10.28 - 16:47.51.945][debug][system] [closed] tcp://192.168.2.178:8080
● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 93 (limit: 11107)
     Memory: 687.9M
        CPU: 17min 34.622s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:50:44 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:50:47 open-haus.lan node[51700]: [2024.10.28 - 16:50.47.495][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:50:47 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:50:47 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:50:47 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:50:47 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:50:47 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:50:47 open-haus.lan node[51700]:   port: 8080
Oct 28 16:50:47 open-haus.lan node[51700]: }
Oct 28 16:50:47 open-haus.lan node[51700]: [2024.10.28 - 16:50.47.497][debug][system] [closed] tcp://192.168.2.178:8080
● connector.service - OpenHaus SmartHome/IoT connector
     Loaded: loaded (/usr/lib/systemd/system/connector.service; enabled; preset: disabled)
     Active: active (running) since Sat 2024-10-26 12:46:59 CEST; 2 days ago
       Docs: https://open-haus.io
   Main PID: 51700 (node)
      Tasks: 102 (limit: 11107)
     Memory: 768.2M
        CPU: 17min 40.018s
     CGroup: /system.slice/connector.service
             └─51700 /usr/bin/node index.js

Oct 28 16:56:35 open-haus.lan node[51700]: Worker spawend for url http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:56:38 open-haus.lan node[51700]: [2024.10.28 - 16:56.38.535][error][system] [error] tcp://192.168.2.178:8080 Error: connect EHOSTUNREACH 192.168.2.178:8080
Oct 28 16:56:38 open-haus.lan node[51700]:     at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1607:16) {
Oct 28 16:56:38 open-haus.lan node[51700]:   errno: -113,
Oct 28 16:56:38 open-haus.lan node[51700]:   code: 'EHOSTUNREACH',
Oct 28 16:56:38 open-haus.lan node[51700]:   syscall: 'connect',
Oct 28 16:56:38 open-haus.lan node[51700]:   address: '192.168.2.178',
Oct 28 16:56:38 open-haus.lan node[51700]:   port: 8080
Oct 28 16:56:38 open-haus.lan node[51700]: }
Oct 28 16:56:38 open-haus.lan node[51700]: [2024.10.28 - 16:56.38.537][debug][system] [closed] tcp://192.168.2.178:8080

start to watch:
16:23 = 49
16:25 = 51
16:28 = 54
16:33 = 63
16:37 = 69

@mStirner mStirner added bug Something isn't working investigate labels Oct 28, 2024
@mStirner
Copy link
Member Author

After restarting the backend, tasks down to 34 and the journal is flooded with disconnects:

Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e567f84469378713d4ad1f/interfaces/66e567f84469378713d4ad21
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.640][debug][forwarder] Woker "ssdp.js" exited: 1006
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.640][warn][forwarder] Worker exited with error, respawn
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a14572e58e7085f2a2e3/interfaces/66e4a14572e58e7085f2a2e4
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: Worker exited with code 0: http://127.0.0.1:8080/api/devices/66e4a0de72e58e7085f2a2db/interfaces/66e4a0de72e58e7085f2a2dc
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.644][debug][forwarder] Woker "mdns.js" exited: 1006
Oct 28 16:57:55 open-haus.lan node[51700]: [2024.10.28 - 16:57.55.644][warn][forwarder] Worker exited with error, respawn

@mStirner
Copy link
Member Author

A quick fix would be a process.exit(1) in the error listener:
sockets/tcp.js

const net = require("net");

module.exports = ({ host, port }) => {

    const logger = require("../system/logger.js");

    let socket = new net.Socket();

    // keep socket alive
    socket.setKeepAlive(true, 5000);
    //socket.unref(); // test for #62, did nothing!

    socket.on("error", (err) => {
        logger.error(`[error] tcp://${host}:${port}`, err);
        // why not cal process.exit() here? 
        // this file/function should only be called in a worker thread
        // so it dosnt matter if the process exits or not...
        // if the socket.unref() approach fails, try the exit method
        process.exit(1);
    });

    socket.on("close", () => {
        logger.debug(`[closed] tcp://${host}:${port}`);
    });

    socket.on("connect", () => {
        logger.info(`[connected] tcp://${host}:${port}`);
    });

    socket.connect(port, host);

    return socket;

};

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working investigate
Projects
None yet
Development

No branches or pull requests

1 participant