Node-RED-docker causes regular CPU spikes

[blagae]

I have recently noticed that a closely monitored docker container using image nodered/node-red:latest (2.1.3) shows, on the processor core (or thread ?) it uses, CPU spikes up to 100% every 30 minutes. My monitoring happens by polling docker stats regularly and graphing it over time. This also occurs if the container is fresh, i.e. with no active flows, no third-party libraries installed, no custom code, etc.

A fuller writeup, with many helpful comments from community members, is available on the Node-RED Forum. When reading through it, bear in mind that the suspicion that Node-RED crashed my system was false.

A quick summary of the salient findings:

• other Node.js containers on my system do not show these spikes.
• manually inserted GC calls do not mitigate the issue, making it unlikely to be related to Node.js GC
• other users can reproduce spikes, whether they're on Docker or not

Given the replies and confirmations that the issue was reproducible, I believe that my environment is irrelevant, but here goes: I am running on a system with an Intel Celeron N3350 processor with 4GB of memory. The OS is Ubuntu 20.04 LTS and I am running Docker version 19.03.8, build afacb8b7f0.

[MichaelLeeHobbs]

The output of sudo journalctl -t dockerd -t docker -S "1 day ago" would be helpful to rule out a docker related issue.

[blagae]

No dice ...

blagae@thingamajig:~$ sudo journalctl -t dockerd -t docker -S "1 day ago"
-- Logs begin at Thu 2021-10-21 18:40:44 CEST, end at Mon 2021-12-06 09:24:02 CET. --
-- No entries --
blagae@thingamajig:~$

[MichaelLeeHobbs]

That's interesting. I assume you have been running docker and the container over the last day but please confirm. Secondly, what flavor of Linux? I've mostly worked with Docker on Fedora based Linux and Windows (in a dev context). There could be a different command we need to run to check the logs. I'm honestly expecting a Docker issue here as I've seen a lot of issues like this but not as much in the last year. You might want to start with updating Docker. Version 19.03.8 came out in March of 2020.

I'm not saying this isn't a Node-Red issue but we have been running Node-Red in prod for years and haven't seen this issue but we have some serious hardware and Node-Red is barely a drop in the bucket compared to some of the other containers we run.

[blagae]

I have indeed been running the container continuously for over a month.

blagae@thingamajig:~$ cat /etc/issue
Ubuntu 20.04 LTS \n \l
blagae@thingamajig:~$ docker --version
Docker version 19.03.8, build afacb8b7f0

In the forum thread linked above, I have written confirmation from other users that they can reproduce the issue as well on their systems (with or without Docker), as well as my own confirmation that it happens on my Windows machine (i.e. non-dockerized).

[MichaelLeeHobbs]

In the forum thread linked above, I have written confirmation from other users that they can reproduce the issue as well on their systems (with or without Docker), as well as my own confirmation that it happens on my Windows machine (i.e. non-dockerized).

Then this should be reported to https://github.com/node-red/node-red/issues and not here.

[hardillb]

Leave it here, we can move it later if needed.

What we do need is an approach to profiling the app in a way that we can easily link up the timestamps of the spikes with the actual code paths.

[thalesmaoa]

Hi @MichaelLeeHobbs , since you've reported to be really common in the past, I decided to post my problem here as well.

I'm facing really high CPU usage on a RPi3. Please, check my docker-compose bellow:

services:
  softplc-nodered:
    image: nodered/node-red
    container_name: softplc-nodered
    restart: always
    group_add: 
      - dialout
    devices:
      - "/dev/ttyAMA0:/dev/ttyAMA0"
    ports:
      - '1880:1880'
    volumes:
      - /home/pi/docker/softplc-nodered/data:/data
    environment:
    environment:
      - TZ=America/Sao_Paulo
      - NODE_OPTIONS="--max_old_space_size=256"

I do have another Pi running Nodered outside a docker with smoother results. How can I debug it? Just to point out, I do prefer run it inside docker.