Parallel tests doesn't run correctly using docker-compose

[OndrejKu]

I'm running my env in docker-compose. I can prepare the database, I see 8 additional databases were created and schema was loaded.
Then I run this:

docker-compose -f ../docker-compose-local.yml run --rm cms rake parallel:spec

and what I see is 8 running specs as expected, but sequentially not in parallel. Also the runtime is higher then just using simple rspec.

[grosser]

need to assign more cpus to docker container, not sure how though

[noefroidevaux]

I have noticed the same behavior. Have you found a solution @OndrejKu ?

[Brad-FDI]

We see this too. Specifically: Docker Desktop on OSX configured with 8 CPUs, tests create 8 databases as expected, but somehow the tests are serialized and the total runtime is slower: 45 mins without parallel, 60 mins with parallel. In our case, the PostGres DB is running on the host machine, maybe it is related -- I kind of wonder if the DB calls out of the container are all being serialized.

[noefroidevaux]

We see this too. Specifically: Docker Desktop on OSX configured with 8 CPUs, tests create 8 databases as expected, but somehow the tests are serialized and the total runtime is slower: 45 mins without parallel, 60 mins with parallel. In our case, the PostGres DB is running on the host machine, maybe it is related -- I kind of wonder if the DB calls out of the container are all being serialized.

I also use Docker Desktop on macOS... but I have a specific container for the DB and I have the same behaviour.

[noefroidevaux]

need to assign more cpus to docker container, not sure how though

@grosser I checked this and it's already the case. In my case, I use Docker Desktop on macOS which by default uses half of the host cpus (5 in my case). By default, these 5 cpus can be used without limit by any container [1]. So I don't think it's the problem.

[1] https://docs.docker.com/config/containers/resource_constraints/ => "By default, each container’s access to the host machine’s CPU cycles is unlimited."

docker info | grep -iE "CPUs|Memory"
 CPUs: 5
 Total Memory: 5.8GiB

[grosser]

run with rake parallel:spec[8] ?
... or is the bug that it does not detect cpu count inside of docker, that should be easy to reproduce ... maybe hard to fix 🤞

[noefroidevaux]

run with rake parallel:spec[8] ?
... or is the bug that it does not detect cpu count inside of docker, that should be easy to reproduce ... maybe hard to fix 🤞

On my side, with rake parallel:spec[5] it's exactly le same as rake parallel:spec. I also tested to run rake parallel:spec on the host and keep the containers for postgres/redis/... and I have almost the same behaviour as I run it without Docker. It seems something happen only in the container where we launch the command (not related to the others containers).

[Brad-FDI]

We are seeing that the serialization/slow-down is only happening on our M1 MacBooks ... tested on 3 different Intel MacBooks and the parallelization inside of Docker Desktop for Mac works great on those. Whereas 3 different M1 MacBooks experience the terrible performance. So we are suspecting the problem is somehow related to Docker Desktop on M1 chips.

[mezbahalam]

Hi @Brad-FDI, I am encountering similar issue and have been trying to find a solution, but haven't had any luck yet. I understand this is an ongoing issue with Docker on M1 chips. Have you been able to find a workaround or solution that improves Docker's performance in this situation? Any suggestions or insights would be greatly appreciated. Thanks!

[Brad-FDI]

Hi @mezbahalam, unfortunately we have no workaround, we're just relying on cloud CI more where parallel_tests works well.