Proposal: Stop shipping grsecurity kernels (on server)

[legoktm]

Proposal: Stop shipping grsecurity kernels (on server)

Affected components

• SecureDrop Server

People and roles

• @legoktm, @lsd-cat (proposers)
• SD team (consulting)

Problem Statement

The grsecurity kernel presents two major problems for us:

• limits hardware compatibility and requires manual testing for each NUC we support (now automated), instead of relying on distro kernels that receive broad testing
• requires manual effort and coordination by SD team members to produce new kernel builds, which means that we're unable to keep up with the pace of upstream releases

Brief security perspective:

• We are currently not using the RBAC policy system¹, which is one of the main useful features of grsec. Writing a meaningful policy for the server now would require significant time from the team that we cannot allocate now.
• We are seemengly using PaX², but the policy we are using is a generic desktop one with rules for Java, browsers, Steam(?)³

My impression is that we are using grsec, mostly for the generic binary exploitation protections, but without leveraging seriously any of the in-depth security features it can provide. Note that we are also using AppArmor (and that reviewing those policies routinely is also an overhead).

Furthermore, our main attack surface remote-wise is the Tor stack, the generic Linux network stack, apache2 and the source interface web application. While we'd like to make privilege escalation difficult, our focus is remote defense.

Solution impact

• We no longer ship grsecurity kernels for the server (we'll keep shipping them in the workstation)
• No significant security holes opened

Requirements and constraints

• Have a rough idea of what protections grsecurity is giving us
• Implement any desired protections/mitigations in other ways
• Identify ideal time to implement the migration (e.g. dist-upgrade)

Exploration

Initial proposal

Selected proposal

Footnotes

1. https://en.wikibooks.org/wiki/Grsecurity/The_RBAC_System ↩

2. https://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities#Controlling_PaX_Flags_(paxctl) ↩

3. https://github.com/freedomofpress/kernel-builder/blob/main/debian/securedrop-grsec/opt/securedrop/paxctld.conf ↩

[zenmonkeykstop]

I have some issues with this proposal, and am not in favour of it in its current form:

Problem Statement

The grsecurity kernel presents two major problems for us:

* limits hardware compatibility and requires manual testing for each NUC we support, instead of relying on distro kernels that receive broad testing

Kernel testing is now automated for supported hardware, so this is no longer a major time sink for the team. While we do recommend specific hardware to reduce support burden and the element of surprise, in practice the grsec kernels run on a wide range of hardware without issues.

* requires manual effort and coordination by SD team members to produce new kernel builds, which means that we're unable to keep up with the pace of upstream releases

Again - in practice the usual impact here is the time it takes to do a single kernel build and step through the release per month, if we stick to a monthly schedule.

When kernel major or minor versions change, the impact is higher, as kernel config or boot option changes may be required. The most extreme case historically was when the kernel version did not have support for the latest Intel networking chipset, requiring a patch to add device numbers. If we don't lag behind on kernel versions this is unlikely to be a problem again.

Brief security perspective:

* We are currently not using the RBAC policy system[1](#user-content-fn-1-78ad2fdf12b3f9a8e54c305eb1e7853d), which is one of the main useful features of grsec. Writing a meaningful policy for the server now would require significant time from the team that we cannot allocate now.

We've never seriously looked at creating said policy, so we don't have a clear idea of the effort involved - even if it was significant it might be worth doing (either by the team or via a 3rd party) given the additional protections it would provide.

* We are seemengly using PaX[2](#user-content-fn-2-78ad2fdf12b3f9a8e54c305eb1e7853d), but the policy we are using is a generic desktop one with rules for Java, browsers, Steam(?)[3](#user-content-fn-3-78ad2fdf12b3f9a8e54c305eb1e7853d)

Yup, this is pretty much the paxctld.conf that ships with the .deb - we could definitely be adding more specific rules. IMO this is also something we should actually look at in terms of resource allocation rather than dismissing out of hand.

My impression is that we are using grsec, mostly for the generic binary exploitation protections, but without leveraging seriously any of the in-depth security features it can provide. Note that we are also using AppArmor (and that reviewing those policies routinely is also an overhead).

It's my impression (correct me if I'm wrong) that Apparmor is providing us with a different kind of protection, via file-based profiles for specific executables (apache2 and tor on the app server, for example). Whereas grsec's protections apply systemwide.

Furthermore, our main attack surface remote-wise is the Tor stack, the generic Linux network stack, apache2 and the source interface web application. While we'd like to make privilege escalation difficult, our focus is remote defense.

Right now we get said generic protections at a low cost in terms of team effort, and as is pointed out above we could leverage other grsecurity features to get even more.

Solution impact

* We no longer ship grsecurity kernels for the server (we'll keep shipping them in the workstation)

This puts us on the hook to keep maintaining the kernel build and release process, so we don't get as much benefit in terms of team resources. It is less likely that we'll see kernel incompatibilities on the workstation, but we'd still have to verify that with major/minor version increments.

* No significant security holes opened

IMO, this isn't demonstrated within this proposal.

Requirements and constraints

* Have a rough idea of what protections grsecurity is giving us

* Implement any desired protections/mitigations in other ways

* Identify ideal time to implement the migration (e.g. dist-upgrade)

If we're gonna put more effort into implementing said protections, why would we not just write RBAC profiles and update paxctld.conf? Changes to the latter at least could most likely be shipped with an updated grsec metapackage as normal.

Exploration

Initial proposal

Selected proposal

Footnotes

1. https://en.wikibooks.org/wiki/Grsecurity/The_RBAC_System [↩](#user-content-fnref-1-78ad2fdf12b3f9a8e54c305eb1e7853d)

2. https://en.wikibooks.org/wiki/Grsecurity/Additional_Utilities#Controlling_PaX_Flags_(paxctl) [↩](#user-content-fnref-2-78ad2fdf12b3f9a8e54c305eb1e7853d)

3. https://github.com/freedomofpress/kernel-builder/blob/main/debian/securedrop-grsec/opt/securedrop/paxctld.conf [↩](#user-content-fnref-3-78ad2fdf12b3f9a8e54c305eb1e7853d)

[legoktm]

Broadly I understand where you're coming from, but my general feeling to most of the things you've pointed out is that yes, there's plenty we could do and ideally would do, but we don't have the capacity nor skill to do so (IMO kernel maintenance is not and shouldn't be a core competency/responsibility of our team). Shipping a custom kernel is always going to have pros/cons and right now IMO we're leaning more towards the cons.

Some specific replies:

Kernel testing is now automated for supported hardware, so this is no longer a major time sink for the team. While we do recommend specific hardware to reduce support burden and the element of surprise,

Oops, yeah. Fixed.

in practice the grsec kernels run on a wide range of hardware without issues.

freedomofpress/kernel-builder#46 is one public example that comes to mind; anecdotally I remember other issues just related to kernels but we should be able to get more specific data (maybe we can look at old support tickets?).

Again - in practice the usual impact here is the time it takes to do a single kernel build and step through the release per month, if we stick to a monthly schedule.

First, monthly releases is already a compromise, because we know we can't keep up with a weekly cycle. This means we are shipping kernels with known public vulnerabilities that are fixed upstream.

But even then we're struggling to keep up with the monthly schedule. We've done it sometimes (longest being a 4-month stretch), but haven't consistently been able to keep pace.

• 2022-11: 5.15.74
• 2022-12: 5.15.81
• 2023-01: 5.15.89
• 2023-07: 5.15.120
• 2023-08: 5.15.123
• 2023-09: 5.15.131
• 2023-10: 5.15.135
• 2023-12: 5.15.140
• 2024-03: 5.15.148
• 2024-04: 5.15.152
• 2024-06: 5.15.160

• No significant security holes opened
  IMO, this isn't demonstrated within this proposal.

Indeed, it needs further research. That's why the "Exploration" and "Initial proposal" sections are still blank.

• Implement any desired protections/mitigations in other ways
  If we're gonna put more effort into implementing said protections, why would we not just write RBAC profiles and update paxctld.conf?

By "implement", I was thinking like, toggle the correct sysctl knobs. The difference of course is that the latter still requires us maintaining a custom kernel.

[rocodes]

Thanks for filing and for the comments so far.

Since this proposal is co-signed by 2 team members with security and kernel-relevant expertise, and is maybe controversial (I at least don't have all the info yet that I need to know how I feel about it, and tbh my initial response was concern but I also respect the opinions of the ppl who proposed this), I propose we make some time to talk about this instead of trying to hash it out only in the ticket. I suspect that the proposers have a lot more they could say than just what's written up here, and maybe it would help us either make a decision, make a smaller-scoped decision and revisit the larger issue later, or figure out what information we'd need to confidently do either of those.

I already see some problem statements that are bundled in (Can we keep up with an ideal kernel-building cadence and do we agree on what that is? How much effort does it take to do so now that we have automated testing? What should our paxctld.conf file look like? Are we clear on what benefits we get from the grsec kernel and how that differs on the workstation and the server?) and being on the same page on all those will help us out.

[lsd-cat]

I was re-reading recent kernel vulnerabilities and techniques, namely

• https://yanglingxi1993.github.io/dirty_pagetable/dirty_pagetable.html
• https://pwning.tech/nftables/

Both very interesting reads, both targeting mostly temporal (rather than spatial) vulnerabilities (double free/UAF), both targeted at hardened kernels (the Pixel 7, and Google's KernelCTF). In both cases, it seems most mitigations are ineffective. I could not yet find particular evidence whether grsec would prevent these techniques or not (wild guess from me would be no given my very limited understanding).

[zenmonkeykstop]

I was re-reading recent kernel vulnerabilities and techniques, namely

* https://yanglingxi1993.github.io/dirty_pagetable/dirty_pagetable.html

* https://pwning.tech/nftables/

Both very interesting reads, both targeting mostly temporal (rather than spatial) vulnerabilities (double free/UAF), both targeted at hardened kernels (the Pixel 7, and Google's KernelCTF). In both cases, it seems most mitigations are ineffective. I could not yet find particular evidence whether grsec would prevent these techniques or not (wild guess from me would be no given my very limited understanding).

By the looks of things, Grsecurity was working on UAF type exploits at least two years ago: https://grsecurity.net/exploiting_and_defending_against_same_type_object_reuse

Whether that would defend against this particular vuln I cannot say.