General discussion

[Bleuzen]

No description provided.

[Bleuzen]

@mmikowski

Initial testing showed the packages were marked as auto, however, neither unattended upgrades nor sudo apt autoremove -y removed them. We found manual installed of earlier kernels including 3 of the same series added them to /etc/apt/apt.conf.d/01autoremove-kernels under APT::NeverAutoRemove.
Test using sudo unattended-upgrades --dry-run --debug

Ubuntu seems to keep some kernels from being autoremoved by default. From /etc/kernel/postinst.d/apt-auto-removal:

This script generates /etc/apt/apt.conf.d/01autoremove-kernels, which contains the kernel packages that should be kept:

At least that is on Linux Mint / Ubuntu 20.04. However something changed here with newer Ubuntu versions.
On Kubuntu 22.04, these files look a lot simpler:

Idk. yet how this is supposed to work. Only the last installed kernel is saved on 22.04 but still autoremove seems to keep more than that.

@mmikowski which version of *Ubuntu are you using + how many kernels do you have installed for testing?

[mmikowski]

@Bleuzen @eeickmeyer spent the day testing.

Thank you for the great notes above. We are running 20.04 and 22.04, although current development is on 20.04. We are using 2-4 kernels for testing (oem, hwe, earlier versions of these). /boot is the default, 705MiB. See note below about raising this to 2.0GiB.

Analysis of unattended-upgrades in 20.04

1. It protects the two highest versions of a any installed kernels. If, however, you install a prior version of a kernel (e.g. sudo apt install linux-image-5.14.0-1008-oem) it protects that too. It appears this protection remains in place until the next kernel install. The installed kernel version is received as the first argument in /etc/kernel/postinst.d/apt-auto-removal.
2. It does not consider or remove kernels and packages where the top-level meta package was removed. For example, sudo apt purge linux-generic-hwe-20.04 does not result in the associated packages being removed even when all of them are marked as auto. This appears as an issue with the autoremove mechanism of apt and in unattended-upgrades. Neither sudo unattended-upgrades --debug --dry-run nor sudo apt autoremove resulted in removal of these files.
3. It does not adjust for, or warn about, available disk space on /boot
4. It does not actually protect the latest 2 versions of a particular meta package like linux-generic-hwe-20.04. Instead, it protects up to 4 blessed kernels: latest-version (the first version in kernel list reverse sorted by version), previous version (2nd entry in sorted list), last installed version (received as $1), and running version (uname -r).
5. It doesn't consider user intent, but uses rigid logic of (4).

Assessment

Our findings show that even if kernel packages are marked auto, deficiencies in the unattended-upgrades and apt autoremove mechanisms result in substantial deficits compared to kernel-cleaner. Most critically, because of its lack of regard for disk space, unattended-upgrades can allow a over-full disk without alerting the user, especially on small /boot partition devices. However, because unattended-upgrades and apt autoremove mechanisms are well-established and have compelling broad infrastructure, they should be the foundation of any long-term and upstream solution.

We should file a bug with 22.04 to bump the /boot partition to 2.0GiB because the Ubuntu 20.04 and 22.04 /boot partition size for encrypted disks is far too small. At just 705M, it can handle only 3 concurrent kernels. We would like to allow for up to 3 top-level kernel series (e.g. hwe, oem, lowlatency) and keep two versions for each. The disk space requirement is therefore 3 x 2 x 200MiB = 1.2GiB; we should like to round up to 2.0GiB for additional safety.

Proposal for Focal (20.04)

At present, most of the shortcomings above have been addressed (1, 2, 3) or can be easily address (4, 5) by our kernel-cleaner script. To deliver benefits to the user most quickly, we propose to update that tool and continue to use it to refine the logic. We have an good set of automated tests to ensure the logic works as expected.

We can check this in here if you wish. We have already made enhancements for issues 2 and 3. Issue 4 is pretty easy to implement, and issue 5 may be better handled in the next proposal and is discussed below.

Proposal for Jammy (22.04)

• Update logic in /etc/kernel/postinst.d/apt-auto-removal script. We propose to use the same heuristics developed by the kernel-cleaner script and work with upstream to get this merged.
• Either fix autoremove or force remnant deletes. When top-level kernel packages that have been removed, the kernel components ('remnants') like images, tools, headers all stay installed even when they are marked auto. These should be autoremovable, and actively removed by unattended-upgrades (this is already in kernel-cleaner).
• Provide for user config to feed selection logic. We envision a GUI where the user can select the desired kernels to keep (hwe, oem, lowlatency, or custom metapackage name), and then the system will auto-purge any that don't match the top two versions of any selected.
• This would keep the auto-mark component from V2, at least until the packagekit mark-auto bug gets fixed upstream. Important: the apt.conf file should probably be renamed to 000-mark-auto or similar, as we want auto applied before and subsequent processing.

I hope that is helpful!

[mmikowski]

Hi @Bleuzen!

Have you had a chance to review the above? What do you think? Ultimately, for 22.04 I think unattended-upgrades should do the following:

• Consider available space in calculations
• Honor user intent by keeping the kernel flavors they specify but remove all other kernel tools, headers, images, modules, etc.
• Keep the latest and penultimate version of each flavor if space allows

There have been a couple advancements for 22.04 which should be useful for this effort:

• It appears we will see the /boot partition bumped to 2.0GB for 22.04. This issue is also addressed by this bug where we have also commented. While we ship with this already increased, it would be good to benefit everyone.
• PackageKit has been fixed fo 22.04+ by Matthias Klumpp ximion.

[mmikowski]

@Bleuzen are you still interested in this? I can add the kernel cleaner script for collaboration here as well if you like. Please let me know!

I think this is a very important project which I'd like to see adopted up-stream. Here is a DFMEA assessment which discusses how catastrophic this problem can be for even advanced users.

[Bleuzen]

@Bleuzen are you still interested in this?

Sure, just have limited time and energy currently.

Also note that my situation / motivation for this may be a little different. I don't use unattended-upgrades or oem kernels and only care about 22.04 as I already upgraded our machines here.

I can add the kernel cleaner script for collaboration here as well if you like.

Yes this would be a nice thing to have :)

I think this is a very important project which I'd like to see adopted up-stream.

Agree, things like this should work ootb.

[mmikowski]

Hi @Bleuzen:

Great, I'll add the script and the tests, probably tomorrow. 22.04 is a good target. With some luck, we see the package kit fix soon.

Discover does not clean up any auto installed packages AFAIK. Unattended upgrades does, but it's logic appears a bit mysterious.

If you are maintaining a number of machines, you might want to use the mark-auto script from above, unattended upgrades (default config) and kernel cleaner script until you are comfortable that unattended upgrades works well enough on its own. If you want to discuss that some time, let me know.

Cheers, Mike

[mmikowski]

@Bleuzen Sorry this took a while, but I've added the script and unit tests. I disentangled the code from the common libs we used to keep it as straight-forward as possible. We can bring this back with the GUI as we progress.

One thing we'd really like to add is the ability for users to select the desired kernel flavors (e.g. oem, hwe, lowlatency) and then return only the last two versions of those. This should be fairly easy.