create update process/solution for cosign key rotation

[bsherman]

Given the urgent cosign key rotation which happened on the morning of July 2, 2024 (9:59AM EDT, specifically), we need a solution to handle upgrades since new images will not be signed with the key which is expected in our policy.

Pre-requisites for any solution:

1. Depends: update all cosign public key files/references in ublue-os org repos #599 - we need the new cosign key in our repos, etc before building new images
2. rebuild all ublue-os organization images (see list of repos in update all cosign public key files/references in ublue-os org repos #599)

Current proposed solutions:

1. write a script to rebase to unverified then rebase back to verified for whatever image is booted
   • pros: simple and will definitely work
   • cons: expensive in terms of user time and bandwidth and a multistep reboot script/process
2. write a script to remove container policy and upgrade
   • pros: simpler and faster from a user perspective
   • cons: need to verify that removing the containers policy does not prevent rpm-ostree from restoring it on upgrade

[p5]

For option 2, We can add a simple tmpfiles.d to ublue-os/main which restores the org's default policy.json from /usr/etc/containers/policy.json on boot.

This can either be used to add the file if no other file exists, or overwrite an existing file depending on the behaviour we want.

A tmpfiles that overwrites the existing policy would look something like this (would need to double check the file permissions):

C+   /etc/containers/policy.json    -   -   -   -   /usr/etc/containers/policy.json

So if we needed to have a script that uploads the new pub key to /etc, changes the policy and rebases, we can do so

[EyeCantCU]

I edited the Ublue entry from our container policy to point to a new public key, updated to an impacted Bazzite image, deleted the container policy, and rebooted. The file was not restored in /etc, but copying it from /usr/etc back to /etc worked.

Working on a script for this right now. Will only impact people using a custom policy.json outside of Ublue. People can stay on their signed image without a rebase. Will need to type in password to run with escalated privileges. No service needed and it's a one time thing

[EyeCantCU]

This should do the trick:

#!/usr/bin/bash

# Fetch new public key 
curl https://raw.githubusercontent.com/ublue-os/bluefin/7964eae79dc188729ba2e890d3570d488a85b5c9/cosign.pub > /etc/pki/containers/ublue-os.pub

# Update key path
jq '.transports.docker."ghcr.io/ublue-os"[0].keyPath = "/etc/pki/containers/ublue-os.pub"' /etc/containers/policy.json > /etc/containers/policy.json

# Upgrade
rpm-ostree upgrade 

# Restore policy
cp /usr/etc/containers/policy.json /etc/containers/policy.json

[noelmiller]

@EyeCantCU Does the restore policy come after the reboot?

[EyeCantCU]

@EyeCantCU Does the restore policy come after the reboot?

The policy only needs to be changed during the upgrade. It can be ran all at once. We need to be sure we're using the new public key in config and have rebuilt the world with it so that that key gets used from then on

[travier]

#600 (comment) looks good.

I would recommend having the container policy config point to keys in /etc instead of /usr to make updates easier in the future.

[p5]

Slightly modified version of @EyeCantCU's script to solve a couple issues:

#!/usr/bin/bash

set -eux

# Fetch new public key
curl https://raw.githubusercontent.com/ublue-os/main/main/cosign.pub > /etc/pki/containers/ublue-os.pub

# Update key path
jq '.transports.docker["ghcr.io/ublue-os"][0].keyPath = "/etc/pki/containers/ublue-os.pub"' /usr/etc/containers/policy.json > /etc/containers/policy.json

# Upgrade
rpm-ostree update

# Restore policy
cp /usr/etc/containers/policy.json /etc/containers/policy.json
cp /usr/etc/pki/containers/ublue-os.pub /etc/pki/containers/ublue-os.pub

[antheas]

cp /usr/etc/containers/policy.json /etc/containers/policy.json

The policy file is user modifiable, it is not handled by rpm-ostree and is subject to drift. You cannot replace it with the default because users might have modified the file.

Modifying /etc after an update is undefined behavior and can lead to undefined behavior if the update fails for some reason. Especially if the user is on an older version.

I second @travier on using /etc for the policy file and I would rather that be the default from now on as well.

Id skip the second part of the script and only download the key and run jq. Then tell the users to update normally.

[antheas]

If the command podman image trust works, I would also rather use that instead. As it avoids a bunch of undefined behavior if the user has modified the policy file. However, I think it does not support setting the signedIdentity tag that seems to be required?

"ghcr.io/ublue-os": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "/etc/pki/containers/ublue-os-new.pub",
                    "signedIdentity": {
                        "type": "matchRepository"
                    }
                }
            ],

[EyeCantCU]

/etc has been made the default in our policy from here on out. Retrieving the key and removing /usr from the path will be sufficient

[EyeCantCU]

This will be enough for end users to run:

#!/usr/bin/bash
set -eux

# Fetch new public key
curl https://raw.githubusercontent.com/ublue-os/main/main/cosign.pub > /etc/pki/containers/ublue-os.pub

# Update key path
jq '.transports.docker["ghcr.io/ublue-os"][0].keyPath = "/etc/pki/containers/ublue-os.pub"' /usr/etc/containers/policy.json > /etc/containers/policy.json

# Upgrade
rpm-ostree update

If podman image trust isn't generating the desired config, we might as well just swap the value with jq. It's the same end result and available across all of our images

[p5]

Maybe we should use an intermediary cp step so we can run jq against the existing policy in /etc?

#!/usr/bin/bash
set -eux

# Fetch new public key
curl https://raw.githubusercontent.com/ublue-os/main/main/cosign.pub > /etc/pki/containers/ublue-os.pub

# Create temporary policy file for jq to read from
cp /etc/containers/policy.json /etc/containers/policy-bak.json

# Update key path
jq '.transports.docker["ghcr.io/ublue-os"][0].keyPath = "/etc/pki/containers/ublue-os.pub"' /etc/containers/policy-bak.json > /etc/containers/policy.json

# Clean up
rm -f /etc/containers/policy-bak.json

echo "Please reboot your system"

[EyeCantCU]

It should already be running against the existing policy, unless there's something I'm missing?

[p5]

It should already be running against the existing policy, unless there's something I'm missing?

Sorry I should have listed the changes.

I had to change the jq to read from /usr/etc because otherwise it refused to write to the same file it was reading from.
I may have done something unrelated wrong though, so happy to be corrected

[EyeCantCU]

Oh! I see what you mean. I didn't have this problem earlier but if you encountered it, we should use your workaround. Same end goal. Thank you!

[antheas]

sed can do in-place writes as well, so that is also an option

[bsherman]

@EyeCantCU @p5 @antheas as a response to both your recent comments, here's my proposal:

We'll create the following script in the config repo: fix-key-and-update.sh

#!/usr/bin/bash
#
# This is a tool to provide easy change to the new Universal Blue image signing key, updated July 2, 2024.
# 
# Note: this is required for upgrades to images published after July 1, 2024, and will prevent downgrading 
# to images published before July 2, 2024.
#
set -eux

# Fetch the new public key from ublue-os's github repo, updating the local copy.
curl https://raw.githubusercontent.com/ublue-os/main/main/cosign.pub > /etc/pki/containers/ublue-os.pub

# Ensure the path to the public key matches the local copy location.
sed -i.bak "s#/usr/etc/pki/containers/ublue-os.pub#/etc/pki/containers/ublue-os.pub#" /etc/containers/policy.json

# Update system, respecting new public signing key.
rpm-ostree update

This accomplishes the goals which have already been addressed, and uses minimal number of commands, making it easier to document. Also, this is idempotent, and will not change a user's policy.json should they have modified it in a way which does not meet our assumptions.

[bsherman]

As a follow up... once we publish this, we can have a Discourse document with a TL;DR to just run curl URL | sudo bash - etc... plus a full explanation and link to the script, and explain things. Announcement should link to the doc.

[castrojo]

ublue-os/config#298

[m2Giles]

We have added a redirect at https://fix.universal-blue.org to the script.

That should be short enough for a user to wget or curl.

[castrojo]

Posted: https://universal-blue.discourse.group/t/important-announcement-regarding-system-updates-action-needed/2689/

Thanks ya'll, and again I apologize for this, thank you so much for mobilizing on behalf of the userbase. Oof.

[Malix-Labs]

Very glad to see such measures taken after a mistake, it shows again a will to do things the right way 😄

[bsherman]

Posted: https://universal-blue.discourse.group/t/important-announcement-regarding-system-updates-action-needed/2689/

With our solution for the surprise cosign key rotation completed, this is done.