In starter and starter_dev file permissions don't allow for composer commands after init

[noahwsmith]

After either make starter or make starter_dev the owner of the web directory appears to be root, and the web/sites/default/settings.php file is unwritable, even though every composer command needs to be able to append to it (apparently):

noahs-m2-mbp-4:isle-dc noah$ docker-compose exec -u nginx drupal bash -c "composer require islandora/islandora_mirador"
Info from https://repo.packagist.org: #StandWithUkraine
Using version ^2.2 for islandora/islandora_mirador
./composer.json has been updated
Running composer update islandora/islandora_mirador
Loading composer repositories with package information
Info from https://repo.packagist.org: #StandWithUkraine
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking islandora/islandora_mirador (2.2.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading islandora/islandora_mirador (2.2.1)
  - Installing islandora/islandora_mirador (2.2.1): Extracting archive
Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
Package silex/silex is abandoned, you should avoid using it. Use symfony/flex instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package webmozart/path-util is abandoned, you should avoid using it. Use symfony/filesystem instead.
Generating autoload files
68 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Scaffolding files for islandora/islandora-starter-site:
  - NOTICE Modifying existing file at [web-root]/sites/default/settings.php. Examine the contents and ensure that it came out correctly.
  - Append to [web-root]/sites/default/settings.php from assets/patches/default_settings.txt

Installation failed, reverting ./composer.json and ./composer.lock to their original content.


  [ErrorException]
  file_put_contents(/var/www/drupal/web/sites/default/settings.php): failed to open stream: Permission denied


require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] [<packages>]...

noahs-m2-mbp-4:isle-dc noah$ docker-compose exec drupal bash -c "ls -lah web/sites/default"
total 88K
dr-xr-xr-x    6 root     root         192 Nov  2 17:41 .
drwxr-xr-x    7 root     root         224 Nov  2 17:41 ..
-rw-r--r--    1 root     root        8.1K Nov  2 17:41 default.services.yml
-rw-r--r--    1 root     root       32.1K Nov  2 17:41 default.settings.php
drwxr-xr-x    7 nginx    nginx       4.0K Nov  2 17:43 files
-r--r--r--    1 root     root       33.3K Nov  2 17:42 settings.php
noahs-m2-mbp-4:isle-dc noah$ docker-compose exec drupal bash -c "chown -R nginx:nginx web"
noahs-m2-mbp-4:isle-dc noah$ docker-compose exec -u nginx drupal bash -c "composer require islandora/islandora_mirador"
Info from https://repo.packagist.org: #StandWithUkraine
Using version ^2.2 for islandora/islandora_mirador
./composer.json has been updated
Running composer update islandora/islandora_mirador
Loading composer repositories with package information
Info from https://repo.packagist.org: #StandWithUkraine
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking islandora/islandora_mirador (2.2.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 0 updates, 0 removals
Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
Package silex/silex is abandoned, you should avoid using it. Use symfony/flex instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package webmozart/path-util is abandoned, you should avoid using it. Use symfony/filesystem instead.
Generating autoload files
68 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Scaffolding files for islandora/islandora-starter-site:
  - NOTICE Modifying existing file at [web-root]/sites/default/settings.php. Examine the contents and ensure that it came out correctly.
  - Append to [web-root]/sites/default/settings.php from assets/patches/default_settings.txt

Installation failed, reverting ./composer.json and ./composer.lock to their original content.


  [ErrorException]
  file_put_contents(/var/www/drupal/web/sites/default/settings.php): failed to open stream: Permission denied


require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] [<packages>]...

noahs-m2-mbp-4:isle-dc noah$ docker-compose exec drupal bash -c "ls -lah web/sites/default"
total 88K
dr-xr-xr-x    6 root     root         192 Nov  2 17:41 .
drwxr-xr-x    7 root     root         224 Nov  2 17:41 ..
-rw-r--r--    1 root     root        8.1K Nov  2 17:41 default.services.yml
-rw-r--r--    1 root     root       32.1K Nov  2 17:41 default.settings.php
drwxr-xr-x    7 nginx    nginx       4.0K Nov  2 17:43 files
-r--r--r--    1 root     root       33.3K Nov  2 17:42 settings.php
noahs-m2-mbp-4:isle-dc noah$ docker-compose exec drupal bash -c "chmod u+w web/sites/default/settings.php"
noahs-m2-mbp-4:isle-dc noah$ docker-compose exec -u nginx drupal bash -c "composer require islandora/islandora_mirador"
Info from https://repo.packagist.org: #StandWithUkraine
Using version ^2.2 for islandora/islandora_mirador
./composer.json has been updated
Running composer update islandora/islandora_mirador
Loading composer repositories with package information
Info from https://repo.packagist.org: #StandWithUkraine
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking islandora/islandora_mirador (2.2.1)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 0 updates, 0 removals
Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
Package silex/silex is abandoned, you should avoid using it. Use symfony/flex instead.
Package symfony/debug is abandoned, you should avoid using it. Use symfony/error-handler instead.
Package webmozart/path-util is abandoned, you should avoid using it. Use symfony/filesystem instead.
Generating autoload files
68 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Scaffolding files for islandora/islandora-starter-site:
  - NOTICE Modifying existing file at [web-root]/sites/default/settings.php. Examine the contents and ensure that it came out correctly.
  - Append to [web-root]/sites/default/settings.php from assets/patches/default_settings.txt
noahs-m2-mbp-4:isle-dc noah$

I think if we have the composer install be run as the nginx user this can be avoided. I will also fix the u+w aspect of the settings.php file in my MR.

[noahwsmith]

It turned out that switching to run everything as nginx was going to require a full refactor. I chose to take the simpler route of just addressing the immediate issues with the linked PR.

[joshdentremont]

I'm getting different results on Linux vs OSX

make starter on Linux:

> docker-compose exec drupal with-contenv bash
bash-5.1# ls -la /var/www/drupal/web/sites/default/
total 96
dr-xr-xr-x    3 nginx    nginx         4096 Dec 22 18:00 .
drwxr-xr-x    3 nginx    nginx         4096 Dec 22 17:59 ..
-rw-r--r--    1 nginx    nginx         8277 Dec 22 17:59 default.services.yml
-rw-r--r--    1 nginx    nginx        32904 Dec 22 17:59 default.settings.php
drwxr-xr-x    4 nginx    nginx         4096 Dec 22 18:01 files
-r--r--r--    1 1004     nginx        34051 Dec 22 18:02 settings.php

make starter on OSX:

> docker-compose exec drupal with-contenv bash
bash-5.1# ls -la /var/www/drupal/web/sites/default/
total 88
dr-xr-xr-x    6 nginx    nginx          192 Dec 22 17:45 .
drwxr-xr-x    7 nginx    nginx          224 Dec 22 17:31 ..
-rw-r--r--    1 nginx    nginx         8277 Dec 22 17:31 default.services.yml
-rw-r--r--    1 nginx    nginx        32904 Dec 22 17:31 default.settings.php
drwxr-xr-x    4 nginx    nginx         4096 Dec 22 17:50 files
-r--r--r--    1 nginx    nginx        34048 Dec 22 17:54 settings.php

Then on both I ran docker-compose exec -T drupal with-contenv bash -lc 'composer require "drupal/block_class:^1.3"' which succeeded on the Linux machine but failed on OSX with the following output:

Scaffolding files for drupal/core:
  - Copy [project-root]/.editorconfig from assets/scaffold/files/editorconfig
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
Scaffolding files for islandora/islandora-starter-site:
  - NOTICE Modifying existing file at [web-root]/sites/default/settings.php. Examine the contents and ensure that it came out correctly.
  - Append to [web-root]/sites/default/settings.php from assets/patches/default_settings.txt

Installation failed, reverting ./composer.json and ./composer.lock to their original content.


  [ErrorException]
  file_put_contents(/var/www/drupal/web/sites/default/settings.php): failed to open stream: Operation not permitted

Local permissions for linux:

ll codebase/web/sites/default/
total 96
dr-xr-xr-x 3 _apt     systemd-journal  4096 Dec 22 14:00 .
drwxr-xr-x 3 _apt     systemd-journal  4096 Dec 22 13:59 ..
-rw-r--r-- 1 _apt     systemd-journal  8277 Dec 22 13:59 default.services.yml
-rw-r--r-- 1 _apt     systemd-journal 32904 Dec 22 13:59 default.settings.php
drwxr-xr-x 2 root     root             4096 Dec 22 14:00 files
-r--r--r-- 1 jdentrem systemd-journal 34107 Dec 22 14:04 settings.php

Local permissions for OSX:

ll codebase/web/sites/default
total 168
dr-xr-xr-x@ 6 josh  101    192 22 Dec 13:45 ./
drwxr-xr-x@ 7 josh  101    224 22 Dec 13:31 ../
-rw-r--r--@ 1 josh  101   8277 22 Dec 13:31 default.services.yml
-rw-r--r--@ 1 josh  101  32904 22 Dec 13:31 default.settings.php
drwxr-xr-x  2 josh  101     64 22 Dec 13:45 files/
-r--r--r--@ 1 josh  101  34048 22 Dec 13:54 settings.php

[joshdentremont]

This seems to only be an issue on OSX, but I don't understand why. If I install on Linux then run composer update -W it works and the permissions are changed to:

-rw-r--r--    1 root     root          8542 Jan  9 17:33 default.services.yml
-rw-r--r--    1 root     root         33395 Jan  9 17:33 default.settings.php
drwxr-xr-x    4 nginx    nginx         4096 Jan  9 17:30 files
-r--r--r--    1 1004     nginx        34107 Jan  9 17:33 settings.php

But if I try to run that on my Macbook it just fails and says it couldn't delete default.services.yml, even though I'm running the composer update through the container as root. It seems like composer wants to remove the default files and replace them, but for some reason it can't. I would think that running code inside the container should work the same no matter what the host machine is, but that doesn't seem to be the case.

I also just did a fresh make local and the permissions are the same as make starter, but it does let me run composer updates. Once I run it, the owner and group are not changed to root, so it does still look like there is something different between starter and local, but I don't think it's the permissions that are causing me issues.

It also seems like what I'm seeing is slightly different to what Noah is seeing.

[DonRichards]

Composer can and usually does run as root. When it installs a new module, the permissions will reflect that (root:root) instead of the standard (1000:nginx) or (nginx:nginx). In some situations, the host machine's user ID isn't 1000. And this will cause an issue. Nginx needs to be able to read most of the packages in the vendor directory and Drupal modules and themes composer installed for it to work. I could put this code into a PR and have it triggered just before & after the compose commands are run so that the directories maintain the proper permissions.

.PHONY: set-codebase-owner
.SILENT: set-codebase-owner
## Updates codebase folder to be owned by the host user and nginx group.
set-codebase-owner:
	@echo ""
	@echo "Setting codebase/ folder owner back to $(shell id -u):101"
	sudo find ./codebase -not -user $(shell id -u) -not -path '*/sites/default/files/*' -exec chown $(shell id -u):101 {} \;
	sudo find ./codebase -not -group 101 -not -path '*/sites/default/files/*' -exec chown $(shell id -u):101 {} \;
	@echo "  └─ Done"
	@echo ""

[DonRichards]

PR added #320
See if this corrects the issue.

[joshdentremont]

@DonRichards @noahwsmith @rosiel I think I've found the issue. It seems that root can append to a read only file on some machines, but not others. It's not a starter issue, but it's happening with the starter site because it is trying to append a patch to settings.php (assets/patches/default_settings.txt) which is not happening with make local.

On the linux machine if I bash into the drupal container I can do this:

bash-5.1# whoami
root
bash-5.1# echo 'test' >> settings.php
bash-5.1#

On my Mac I get this:

bash-5.1# whoami
root
bash-5.1# echo 'test' >> settings.php
bash: settings.php: Operation not permitted

What I can't figure out is why. Both have the following permissions:
-r--r--r-- 1 1004 nginx 34609 Jan 24 16:01 settings.php

So the options I can see to fix this are:

1. We can give settings.php write permissions for the owner which solves the problem, but any time we make down and make up again the permissions are set back to read only. (I think this is a Drupal security thing)
2. We could find another way to apply this patch to settings.php
3. We could figure out why root can't append to read only files on OSX and possibly fix that

[joshdentremont]

I've done a bit more digging into this, and the patch for settings.php doesn't actually seem to be necessary for Isle installs. All it is doing is adding this line to settings.php: $settings['config_sync_directory'] = '../config/sync'; but in an Isle install that directory is already set.

So we could probably just remove the scaffolding bit in composer.json and it would be fine. I'm not sure if this would cause issues for installs outside of Isle though.