Brief description
When parsing HTTP/1.x header values, Envoy 1.9 and before does not reject embedded zero characters (NUL, ASCII 0x0). This allows remote attackers crafting header values containing embedded NUL characters to potentially bypass header matching rules, gaining access to unauthorized resources.
CVSS
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L (8.3, High)
Affected version(s)
Envoy 1.9.0 and before.
Affected component(s)
HTTP/1.x codec, HTTP router, external authorization, rate limiting service, access logging, likely others.
Attack vector(s)
Embedded NUL in HTTP/1.x header delivered by untrusted client.
Discover(s)/Credits
Harvey Tuch, Google
Example exploit or proof-of-concept
An example scenario in which a “deny” route rule can be exploited is with suffix matching in a RouteConfiguration:
- An Envoy user makes use of x-foo to do target cluster routing.
- A RouteMatch is configured to disallow the suffix "_special_cluster" in x-foo.
- An untrusted user sends "x-foo: some_special_cluster\0" on the wire.
- The header match logic uses string_view for matching. So, it does not match the sinkhole rule.
- The cluster header routing uses c_str, so will then route to cluster "some_special_cluster".
An example scenario in which “deny” authorization can be exploited with exact matching and ext_authz:
- An Envoy user wants to block content-type with “application/javascript” via ext_authz.
- An untrusted user sends “content-type: application/javascript\0”. The ext_authz filter uses string_view for request header copying.
- The ext_authz server allows the request to continue as there is no match.
- Envoy will copy to the wire the original content-type header, including NUL.
- A backend server may incorrectly treat the content-type as “application/javascript”, if it does not reject NUL. While technically this indicates a defect in the backend server, the backend might be using the same codec as Envoy, http-parser, and hence this defect is correlated with CVE-2019-9900.
An example scenario in which “allow” authorization can be exploited with suffix matching and ext_authz:
- An Envoy users wants to only allow access to hosts ending in “.acme.com”.
- An untrusted user sends “host: cat.pictures.com\0.acme.com” on the wire. The ext_authz filter uses string_view for request header copying.
- The ext_authz filter matches on the suffix “.acme.com” and allows the request to continue.
- Envoy uses c_str for virtual host selection. So, it will route with “cat.pictures.com”.
- As with the above exploit, Envoy will write a host: header on the wire with an embedded NUL. The seriousness will depend on whether the backend server for “cat.pictures.com” rejects the request due to this violation.
Details
Envoy expects that its HTTP codecs (http-parser, nghttp2) enforce RFC constraints on valid header values (https://tools.ietf.org/html/rfc7230#section-3.2.6). In particular, it is expected that there are no embedded NUL characters in paths, header values or keys. This is particularly important because two views of a HeaderString are allowed, via c_str() and getStringView(); embedded NULs result in inconsistent views through these accessors. A mixture of these accessors are used in header matching and routing.
The issue first became noticeable via fuzzers when an explicit ASSERT check for embedded NULs was added in #6170. The following issues were tripped by Envoy’s wire-level fuzzers:
- HTTP/1.x & http-parser via h1_capture_fuzz_test: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=13613
- HTTP/2 & nghttp2 via codec_impl_fuzz_test: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=13614
The HTTP/1.x issue is the most serious, as the HTTP/2 failure does not propagate deeply into Envoy; an invalid HeaderString is formed during the header receive callback but only occurs in some corner cases where the stream is already closed (nghttp2 validates correctly the rest of the time), Envoy ignores these headers and so this is unlikely to impact Envoy.
For HTTP/1.x, the errant behavior is due to a bug in how validation of header values is performed by http-parser. You can see this in the validation logic at https://github.com/nodejs/http-parser/blob/0d0a24e19eb5ba232d2ea8859aba2a7cc6c42bc4/http_parser.c#L1469. Only the first character of the header value is validated at line 1490. Then the entire header value is accepted via a memchr scan at https://github.com/nodejs/http-parser/blob/0d0a24e19eb5ba232d2ea8859aba2a7cc6c42bc4/http_parser.c#L1506.
This means that a remote attacker can introduce embedded NULs in any HTTP/1.x header value.
See the above exploitation examples for how this can be leveraged. There are a number of places that Envoy is inconsistent in how it treats header values when deciding on forwarding action and when executing the forwarding action. In particular, a string_view is used for headers in ext_authz RPCs. c_str() is used not just for routing but also access log output and rate limiting service RPCs.
Whether a particular Envoy configuration is vulnerable is highly dependent on how headers are used for access control and routing. If header based access control or routing is not used, this is probably benign, at worst Envoy emits invalid HTTP/1.1 at the backend. This does not appear to affect path based routing, which is handled via different code in http-parser.
Given the number of places header values are interpreted with different string views (for non-test codes, > 150 c_str() and > 80 getStringView() at a recent snapshot), together with the above example exploit, it appears likely there are many potential ways to leverage this inconsistency to bypass the intended access control and routing configuration set by an operator. While this disclosure is primarily related to access control, it’s conceivable that there are DoS implication. Specifically, an attacker may circumvent DoS prevention systems such as rate limiting and authorization for a given backend server.
Mitigations
One of the most direct potential exploits is via suffix based header matching. Avoiding this both in Envoy and external servers such as ext_authz will avoid one common possible attack, but not all, since an exact match on headers with ext_authz is one of the examples provided above. Prefix match rules should be largely safe, although the backend remains unprotected from this invalid input. Regular expression matches will largely depend on the contents of the regular expression.
External data plane sidecall servers such as ext_authz that interact with Envoy supplied headers can be modified to detect and reject requests with embedded NULs.
Detection
Based on current information, this only affects HTTP/1.1 traffic. If this is not structurally possible in your network or configuration, then it is unlikely that this vulnerability applies.
File-based access logging uses the c_str() representation for header values, as does gRPC access logging, so there will be no trivial detection via Envoy’s access logs by scanning for NUL. Instead, operators might look for inconsistencies in logs between the routing that Envoy performs and the logic intended in the RouteConfiguration.
External authorization and rate limit services can check for NULs in headers. Backend servers might have sufficient logging to detect NULs or unintended access; it’s likely that many will simply reject NULs in this scenario via 400 Bad Request, as per RFC 7230.
References
CVE-2019-9900
Brief description
When parsing HTTP/1.x header values, Envoy 1.9 and before does not reject embedded zero characters (NUL, ASCII 0x0). This allows remote attackers crafting header values containing embedded NUL characters to potentially bypass header matching rules, gaining access to unauthorized resources.
CVSS
CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:L/I:L/A:L (8.3, High)
Affected version(s)
Envoy 1.9.0 and before.
Affected component(s)
HTTP/1.x codec, HTTP router, external authorization, rate limiting service, access logging, likely others.
Attack vector(s)
Embedded NUL in HTTP/1.x header delivered by untrusted client.
Discover(s)/Credits
Harvey Tuch, Google
Example exploit or proof-of-concept
An example scenario in which a “deny” route rule can be exploited is with suffix matching in a RouteConfiguration:
An example scenario in which “deny” authorization can be exploited with exact matching and ext_authz:
An example scenario in which “allow” authorization can be exploited with suffix matching and ext_authz:
Details
Envoy expects that its HTTP codecs (http-parser, nghttp2) enforce RFC constraints on valid header values (https://tools.ietf.org/html/rfc7230#section-3.2.6). In particular, it is expected that there are no embedded NUL characters in paths, header values or keys. This is particularly important because two views of a HeaderString are allowed, via c_str() and getStringView(); embedded NULs result in inconsistent views through these accessors. A mixture of these accessors are used in header matching and routing.
The issue first became noticeable via fuzzers when an explicit ASSERT check for embedded NULs was added in #6170. The following issues were tripped by Envoy’s wire-level fuzzers:
The HTTP/1.x issue is the most serious, as the HTTP/2 failure does not propagate deeply into Envoy; an invalid HeaderString is formed during the header receive callback but only occurs in some corner cases where the stream is already closed (nghttp2 validates correctly the rest of the time), Envoy ignores these headers and so this is unlikely to impact Envoy.
For HTTP/1.x, the errant behavior is due to a bug in how validation of header values is performed by http-parser. You can see this in the validation logic at https://github.com/nodejs/http-parser/blob/0d0a24e19eb5ba232d2ea8859aba2a7cc6c42bc4/http_parser.c#L1469. Only the first character of the header value is validated at line 1490. Then the entire header value is accepted via a memchr scan at https://github.com/nodejs/http-parser/blob/0d0a24e19eb5ba232d2ea8859aba2a7cc6c42bc4/http_parser.c#L1506.
This means that a remote attacker can introduce embedded NULs in any HTTP/1.x header value.
See the above exploitation examples for how this can be leveraged. There are a number of places that Envoy is inconsistent in how it treats header values when deciding on forwarding action and when executing the forwarding action. In particular, a string_view is used for headers in ext_authz RPCs. c_str() is used not just for routing but also access log output and rate limiting service RPCs.
Whether a particular Envoy configuration is vulnerable is highly dependent on how headers are used for access control and routing. If header based access control or routing is not used, this is probably benign, at worst Envoy emits invalid HTTP/1.1 at the backend. This does not appear to affect path based routing, which is handled via different code in http-parser.
Given the number of places header values are interpreted with different string views (for non-test codes, > 150 c_str() and > 80 getStringView() at a recent snapshot), together with the above example exploit, it appears likely there are many potential ways to leverage this inconsistency to bypass the intended access control and routing configuration set by an operator. While this disclosure is primarily related to access control, it’s conceivable that there are DoS implication. Specifically, an attacker may circumvent DoS prevention systems such as rate limiting and authorization for a given backend server.
Mitigations
One of the most direct potential exploits is via suffix based header matching. Avoiding this both in Envoy and external servers such as ext_authz will avoid one common possible attack, but not all, since an exact match on headers with ext_authz is one of the examples provided above. Prefix match rules should be largely safe, although the backend remains unprotected from this invalid input. Regular expression matches will largely depend on the contents of the regular expression.
External data plane sidecall servers such as ext_authz that interact with Envoy supplied headers can be modified to detect and reject requests with embedded NULs.
Detection
Based on current information, this only affects HTTP/1.1 traffic. If this is not structurally possible in your network or configuration, then it is unlikely that this vulnerability applies.
File-based access logging uses the c_str() representation for header values, as does gRPC access logging, so there will be no trivial detection via Envoy’s access logs by scanning for NUL. Instead, operators might look for inconsistencies in logs between the routing that Envoy performs and the logic intended in the RouteConfiguration.
External authorization and rate limit services can check for NULs in headers. Backend servers might have sufficient logging to detect NULs or unintended access; it’s likely that many will simply reject NULs in this scenario via 400 Bad Request, as per RFC 7230.
References