Wrong assumption about $ROOT_DEV is LVM

[d2xdt2]

# pvcreate /dev/vda2 
# vgcreate docker /dev/vda2
# reboot
# echo 'VG=docker' > /etc/sysconfig/docker-storage-setup
# systemctl stop docker
# rm -rf /var/lib/docker
# docker-storage-setup
  Volume group "vda1" not found
  Cannot process volume group vda1

The problem is in

ROOT_DEV=$( awk '$2 ~ /^\/$/ && $1 !~ /rootfs/ { print $1 }' /proc/mounts )
ROOT_VG=$( lvs --noheadings -o vg_name $ROOT_DEV | sed -e 's/^ *//' -e 's/ *$//')

as ROOT_DEV becomes /dev/vda1 which is not LVM, and hence the script breaks.

Why is there an assumption that ROOT_DEV is LVM?

[rhvgoyal]

This script so far assumes that LVM is being used. If we want it to work on systems which does not use LVM and use partitions directly, it will need to be fixed.

[anweiss]

+1... this affects any RHEL VMs deployed to Azure where LVM is not used by the root device

[rhvgoyal]

@anweiss Are you looking for growing root partition or looking to setup thin pool. If you are looking to setup thin pool, it will fail anyway (in default setup) as there is most likely no free space on root disk.

I think best is to add another disk to VM and use that disk to setup lvm thin pool.

[anweiss]

hey @rhvgoyal ...so I get this same error when I attach a disk to the VM and use it to setup the thin pool

[rhvgoyal]

@anweiss Have you tried latest upstream docker-storage-setup. My understanding is that script should give you info message Volume group backing root filesystem could not be determined and continue to setup lvm thin pool.

[anweiss]

@rhvgoyal I have... indeed, it sets up the lvm thin pool, but the docker-storage-setup.service fails to start due to the same error

[rhvgoyal]

@anweiss Can you paste the error. If lvm thin pool has been setup, then script should exit with success instead of failure.

Also above code has changed. So I can't understand that how it fails with same error.

[anweiss]

@rhvgoyal sure thing:

● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Mar 21 15:52:20 osnode02 docker-storage-setup[4730]: Volume group "sda1" not found
Mar 21 15:52:20 osnode02 docker-storage-setup[4730]: Cannot process volume group sda1
Mar 21 15:52:21 osnode02 docker-storage-setup[4730]: /dev/sdc has partitions: sdc1
Mar 21 15:52:21 osnode02 systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Mar 21 15:52:21 osnode02 systemd[1]: Failed to start Docker Storage Setup.
Mar 21 15:52:21 osnode02 systemd[1]: Unit docker-storage-setup.service entered failed state.
Mar 21 15:52:21 osnode02 systemd[1]: docker-storage-setup.service failed.
Mar 21 15:52:21 osnode02 systemd[1]: start request repeated too quickly for docker-storage-setup.service
Mar 21 15:52:21 osnode02 systemd[1]: Failed to start Docker Storage Setup.
Mar 21 15:52:21 osnode02 systemd[1]: docker-storage-setup.service failed.

This occurs after the script is successfully executed and during initial startup of Docker

[rhvgoyal]

I think this is complaining about that /dev/sdc1 has paritions. lvm thin pool should still work though. Are you depending on service exit status somewhere?

One easy workaround is to create a volume group yourself and ask docker-storage-setup to create
lvm thin pool from that volume group. Say you have a device /dev/sdb. Create a volume group docker-vg from this device and then pass this to docker-storage-setup.

VG=docker-vg

And start docker-storage-setup and it should be fine.

[rhvgoyal]

I think docker is restarting for some reason and forcing restart of docker-storage-setup too. Can you disable Restart=on-failure and replace it with Restart=on-abnormal in docker.service file. This should cut down on restart when docker fails to start form some configuration failure etc.

IOW, if your thin pool has been setup once, I think it is there and working and if docker is failing to start, it should be failing to start for some other reason. Do "systemctl status docker -l". Failure exit status of docker-storage-setup is something which can be ignored for this configuration. (Right now it does not have mechanism to keep track of devices it has already processed).

[anweiss]

thanks @rhvgoyal. let me do some digging and report back on this

[rhvgoyal]

@anweiss I think the code you are running might be little old. If you look at upstream code, we detect if device is already part of volume group and in that case we know we partitioned the device in previous run and we only give "INFO" message out.

INFO: Device /dev/vdb is already partitioned and is part of volume group docker-vg

Otherwise we error out with

Device $dev is already partitioned and cannot be added to volume group $VG"