CPU/RAM restrictions are not enforced

[gapan]

I'm trying to set CPU and RAM restrictions for my docker containers but they don't seem to really apply. As far as I can tell, fogify only checks the sum of resources against the host system on startup, instead of actually enforcing them. There are multiple issues with this.

• I can maximize the use of the host CPU from within any container, even if it has a CPU limit set
• I can see and use the total amount of the host RAM from within any container, even if it has a RAM limit set
• I have run containers with limits of 0.1G RAM and 5 MHz (!) CPU. Applications within the container work with no issues at native speed and using as much RAM as they need (obviously more than 0.1G)

This is also connected to #3 I guess.

[MosesSymeonidis]

Actually, Fogify translates its resources model to the constraints at the deploy section of a docker-compose file. Then, Fogify deploys docker-compose file to the docker swarm. For CPU constraints, first of all, you have to specify the host's frequency (environmental parameter CPU_FREQ at the controller). Moreover, you should have in mind that Fogify considers that all nodes of the cluster have the same cores' frequency. Then fogify will translate the CPU the provided cores and frequency that specified at its model with the following formula:

CPU-limit = number_of_emulated_cores * emulated_frequency / host_frequency

CPU-limit is the CPU limit that Fogify will provide at docker-compose file for the respective service.

According to the memory constraints, that is really odd. I know docker has some issues with java containers, and the memory limits but I am not sure if this is the case. Can you provide more information about the service that you would like to run?

[gapan]

OK, so I did more tests and at least with respect to RAM, you're right! It seems I was misguided, because if I run the free command (or htop or similar) from within the running container, I can see the entirety of the host RAM as available. But if I run docker stats, I get something like the following (this was run with a restriction of 0.1GB of RAM per node):

CONTAINER ID   NAME                                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O       PIDS
20611585d06d   fogify_node.2.rj4o4vhnmi5vfnbxggi2xc4j2   0.01%     102.2MiB / 102.4MiB   99.77%    848B / 138B       40MB / 6.15MB   18
ba1b3ad3fab4   fogify_node.1.c1i9n4en8wnvnw6yiwhun84p4   0.04%     102.1MiB / 102.4MiB   99.70%    1.17kB / 491B     563MB / 338MB   68
45a296de37e0   fogify_ycsb.1.cogpqj8jntzkvy3cjg75777jn   0.00%     5.266MiB / 2GiB       0.26%     732B / 0B         7.04MB / 0B     2
f2b0555989f2   fogify-db-benchmarks_agent_1              0.01%     43.11MiB / 62.91GiB   0.07%     3.85MB / 27.6kB   0B / 1.52MB     3
2583d31182bf   fogify-db-benchmarks_ui_1                 0.01%     139.6MiB / 62.91GiB   0.22%     18.3kB / 5.7kB    0B / 0B         27
6b036ed13e6a   fogify-db-benchmarks_controller_1         0.02%     72.18MiB / 62.91GiB   0.11%     10.3kB / 21.9kB   0B / 442kB      3
91e4d516123b   fogify-db-benchmarks_cadvisor_1           0.01%     22.82MiB / 62.91GiB   0.04%     25.1kB / 3.84MB   0B / 0B         20

So it seems to be OK. Although I am still confused how the database I'm running within the container could even start with only 100MB of RAM available. It shouldn't be possible and I'm not certain what docker does when a container reaches its declared limits. But this is not an issue with fogify anyway.

With respect to CPU, it's a bit more complicated I think. Please take a look at this repo: https://github.com/Datalab-AUTH/fogify-db-benchmarks

First, build the "stress" docker image:

cd test-CPU-RAM-restrictions
docker build -t stress .

You can edit the CMD line in the Dockerfile to change the number of CPUs (--cpus) and RAM (--vm-bytes) to match your PC first.

Then you can run the start-CPU-RAM-test.sh script in the main dir:

export CPU_FREQ=13620
./start-CPU-RAM-test.sh

If at the same time I run docker stats, I get something like this:

CONTAINER ID   NAME                                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O        PIDS
b7719ad42e5b   fogify_node.1.7n37311kws92zuhywl9bhjv0j   395.35%   102.1MiB / 102.4MiB   99.74%    508B / 0B         37.5MB / 498MB   9
896d0f79c182   fogify-db-benchmarks_agent_1              11.83%    38.54MiB / 62.91GiB   0.06%     452kB / 8.34kB    0B / 754kB       3
37bd2f64c3f7   fogify-db-benchmarks_cadvisor_1           8.31%     14.47MiB / 62.91GiB   0.02%     6.82kB / 448kB    0B / 0B          20
c4ad3f744564   fogify-db-benchmarks_controller_1         3.87%     72.11MiB / 62.91GiB   0.11%     6.96kB / 7.88kB   0B / 442kB       3
406c75f7b4b4   fogify-db-benchmarks_ui_1                 0.00%     88.54MiB / 62.91GiB   0.14%     6.38kB / 4.15kB   0B / 0B          4

The RAM indeed seems limited, but the CPU seems that isn't (~400% in the first line). If I monitor the host system with something like htop I can also see that all CPUs hit 100%. The limit that I'm using here for the CPU for these tests is a ridiculous 5MHz, so I don't think it should be doing that.

But I understand that there may be nothing that you can do from within fogify and this is probably an issue with docker itself. So feel free to close this if you think so too.