s2a: Add gRPC S2A #11113

rmehta19 · 2024-04-18T17:24:28Z

Add S2A Java client to gRPC Java.

Context: https://github.com/google/s2a-go/blob/main/README.md

linux-foundation-easycla · 2024-04-18T17:24:32Z

The committers listed above are authorized under a signed CLA.

✅ login: rmehta19 / name: Riya Mehta (739ee23, b35f145, 46691df, 2dd1c7e, 72630d8, 217a3e4, 44fe552, f94cc10, 08f8342, f47c560, 1d41d10, 42cd3e8, f96d395, 12586b1, 35084af, 7f26712, d1f413b, 655f0bd, da330cd, 50b7366, 0e059e4, a8cacb0, 3198eec, 3184cdc, 38b0a3a, b021a21, 752627a, 19583a7)

matthewstevenson88 · 2024-04-22T15:56:17Z

Thanks @rmehta19. Can you PTAL at the test failures?

rmehta19 · 2024-04-22T18:34:59Z

@matthewstevenson88, I made 2 changes and it looks like 2/3 linux runs are passing. tests(11) is failing with an error in code that was not affected by this PR, so I don't think that failure is related.

The changes:

The build originally failed because the generated grpc code in this PR did not match the code generated by the codegen plugin. I couldn't easily compile the codegen myself(due to reasons explained in: protoc-gen-grpc-java not available on apple m1 #7690). So instead I used the latest published binary available on https://mvnrepository.com/artifact/io.grpc/protoc-gen-grpc-java (1.63) to generate S2AServiceGrpc.java (Created and used a separate simple maven project just to run the plugin and get grpc gen code). Then I changed line 8 & 9 of S2AServiceGrpc.java to match other generated code in this repo (ex:

grpc-java/alts/src/generated/main/grpc/io/grpc/alts/internal/HandshakerServiceGrpc.java

Lines 8 to 9 in fb9a108

    
           value = "by gRPC proto compiler", 
        
           comments = "Source: grpc/gcp/handshaker.proto")

). This resolved the errors thrown by

grpc-java/.github/workflows/testing.yml

Lines 65 to 66 in 3c867c9

    
           - name: Check for modified codegen 
        
             run: test -z "$(git status --porcelain)" || (git status && echo Error Working directory is not clean. Forget to commit generated files? && false)

A failure in the tests was caused when IntegrationTest.java is in package io.grpc.s2a: GetAuthenticationMechanismsTest.java fails because the flag is not being set by JCommander (or perhaps it is getting overwritten). When I moved IntegrationTest.java into the same package as GetAuthenticationMechanismsTest.java: io.grpc.s2a.handshaker, this error does not happen.

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

s2a/src/main/java/io/grpc/s2a/handshaker/S2ATrustManager.java

s2a/src/main/proto/grpc/gcp/common.proto

matthewstevenson88 · 2024-04-25T21:42:34Z

LGTM, thanks @rmehta19! When you have a chance, please remove "draft" to the PR title and can we say a bit more in the CL description including pointing to the S2A-Go README (similar to what we did in the proto PR)?

After that, can we get sign-off from @erm-g and @ejona86?

rmehta19 · 2024-04-26T20:01:50Z

LGTM, thanks @rmehta19! When you have a chance, please remove "draft" to the PR title and can we say a bit more in the CL description including pointing to the S2A-Go README (similar to what we did in the proto PR)?

After that, can we get sign-off from @erm-g and @ejona86?

Done -- thank you for the review @matthewstevenson88!

s2a/src/main/java/io/grpc/s2a/channel/S2AChannelPool.java

s2a/src/main/java/io/grpc/s2a/channel/S2AGrpcChannelPool.java

s2a/src/main/java/io/grpc/s2a/channel/S2AHandshakerServiceChannel.java

s2a/src/main/java/io/grpc/s2a/handshaker/ConnectionIsClosedException.java

s2a/src/main/java/io/grpc/s2a/handshaker/S2AIdentity.java

s2a/src/main/java/io/grpc/s2a/handshaker/tokenmanager/SingleTokenFetcher.java

s2a/src/test/resources/config.cnf

s2a/src/test/java/io/grpc/s2a/handshaker/GetAuthenticationMechanismsTest.java

ejona86

To make sure I understand, s2a is the public API, and s2a/channel and s2a/handshaker are internal APIs. Are those internal APIs to be used anywhere, even within Google?

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

s2a/src/main/proto/grpc/gcp/common.proto

rmehta19 · 2024-09-09T21:32:19Z

Thanks @larry-safran, the item's have been addressed

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

s2a/src/main/java/io/grpc/s2a/channel/S2AChannelPool.java

s2a/src/main/java/io/grpc/s2a/channel/S2AGrpcChannelPool.java

s2a/src/main/java/io/grpc/s2a/handshaker/GetAuthenticationMechanisms.java

s2a/src/main/java/io/grpc/s2a/handshaker/ProtoUtil.java

s2a/src/main/java/io/grpc/s2a/handshaker/S2AProtocolNegotiatorFactory.java

s2a/src/test/java/io/grpc/s2a/handshaker/IntegrationTest.java

rmehta19 · 2024-09-12T17:40:42Z

Thanks for the review @larry-safran ! Please let me know if there is anything else to address.

…tiator.

s2a/src/test/java/io/grpc/s2a/handshaker/IntegrationTest.java

ejona86

Some random things I noticed.

ejona86 · 2024-09-16T20:28:12Z

s2a/src/main/java/io/grpc/s2a/MtlsToS2AChannelCredentials.java

+ * Configures an {@code S2AChannelCredentials.Builder} instance with credentials used to establish a
+ * connection with the S2A to support talking to the S2A over mTLS.
+ */
+public final class MtlsToS2AChannelCredentials {


Why is this a different builder than S2aChannelCredentials? If it just is for different arguments... that's why we have the Builder.

ejona86 · 2024-09-16T20:33:08Z

s2a/src/main/java/io/grpc/s2a/MtlsToS2AChannelCredentials.java

+      File certChainFile = new File(certChainPath);
+      File trustBundleFile = new File(trustBundlePath);
+
+      AdvancedTlsX509KeyManager keyManager = new AdvancedTlsX509KeyManager();


You don't need AdvancedTlsX509KeyManager here. All you needed was:

TlsChannelCredentials.newBuilder() .keyManager(certChainFile, privateKeyFile) .trustManager(trustBundleFile) .build();

Done in 9c44754

ejona86 · 2024-09-16T20:34:35Z

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

+   * @param s2aAddress the address of the S2A server used to secure the connection.
+   * @return a {@code S2AChannelCredentials.Builder} instance.
+   */
+  public static Builder createBuilder(String s2aAddress) {


This should follow the naming convention of newBuilder()

Done in 467ddf5

ejona86 · 2024-09-16T20:36:25Z

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

+  public static final class Builder {
+    private final String s2aAddress;
+    private ObjectPool<Channel> s2aChannelPool;
+    private Optional<ChannelCredentials> s2aChannelCredentials;


Instead of using Optional, default this to InsecureChannelCredentials.create().

Done in ef53441

ejona86 · 2024-09-16T20:39:54Z

s2a/src/main/java/io/grpc/s2a/channel/S2AHandshakerServiceChannel.java

+    @Override
+    public Channel create() {
+      EventLoopGroup eventLoopGroup =
+          new NioEventLoopGroup(1, new DefaultThreadFactory("S2A channel pool", true));


Why create your own event loop? With ALTS that was necessary because it would block (and it was supposed to be temporary). We should do no blocking with s2a.

Removed the usage of own event loop in 03ced73

ejona86 · 2024-09-16T20:45:32Z

s2a/src/main/java/io/grpc/s2a/channel/S2AHandshakerServiceChannel.java

+ */
+@ThreadSafe
+public final class S2AHandshakerServiceChannel {
+  private static final ConcurrentMap<String, Resource<Channel>> SHARED_RESOURCE_CHANNELS =


I'm uncomfortable with s2aChannelCredentials not being part of the cache key.

Do we even need this global? Having at most one connection per s2a-using Channel doesn't seem that bad. The channel can be configured with idleTimeout() or we tear it down when we aren't doing any handshakes.

This was forked from alts, but we don't need it for S2A. I've removed the usage of the global in df413b1

ejona86 · 2024-09-16T20:47:07Z

s2a/src/main/java/io/grpc/s2a/channel/S2AHandshakerServiceChannel.java

+      try {
+        isDelegateTerminated =
+            delegate.awaitTermination(DELEGATE_TERMINATION_TIMEOUT.getSeconds(), SECONDS);
+      } catch (InterruptedException e) {


Missing Thread.currentThread().interrupt()

Done in c1375bb

ejona86 · 2024-09-16T20:48:42Z

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java

+ * Configures gRPC to use S2A for transport security when establishing a secure channel. Only for
+ * use on the client side of a gRPC connection.
+ */
+public final class S2AChannelCredentials {


All the public APIs should be labeled with @ExperimentalApi. The string is a tracking issue; just creating one for all the APIs is fine.

Done in 2a169ba

ejona86 · 2024-09-16T20:52:19Z

s2a/src/main/java/io/grpc/s2a/handshaker/S2ATrustManager.java

+    SessionResp resp;
+    try {
+      resp = stub.send(reqBuilder.build());
+    } catch (IOException | InterruptedException e) {


Missing Thread.currentThread().interrupt()

Done in c1375bb

ejona86 · 2024-09-16T20:55:54Z

s2a/src/main/java/io/grpc/s2a/handshaker/S2AProtocolNegotiatorFactory.java

+            @Override
+            public void onSuccess(SslContext sslContext) {
+              ChannelHandler handler =
+                  InternalProtocolNegotiators.tls(sslContext).newHandler(grpcHandler);


Since you are blocking in the S2ATrustManager, you must use tls(SslContext, ObjectPool<Executor>) for SSLHandler to avoid blocking on the event loop.

Thanks for pointing this out, although in order to do this, I think we need to add a new accessor to

grpc-java/netty/src/main/java/io/grpc/netty/ProtocolNegotiators.java

Line 685 in 9b0c19e

public static ProtocolNegotiator tls(SslContext sslContext,

from InternalProtocolNegotiators.

Changes + adding the new accessor done in: 03ced73

ejona86

More random stuff. This is mostly from just skimming randomly and noticing unexpected code shapes.

ejona86 · 2024-09-16T23:02:33Z

s2a/src/test/java/io/grpc/s2a/handshaker/IntegrationTest.java

+public final class IntegrationTest {
+  private static final Logger logger = Logger.getLogger(FakeS2AServer.class.getName());
+
+  private static final String CERT_CHAIN =


Do they have an expiration date? In all other cases we have documentation on how these were created so we 1) know what they are and 2) know how to recreate them. FakeWriter has even more...

I have removed the hardcoded certs from this file and FakeWriter. I changed these files to read the certs from the resources folder like the rest of the tests. I regenerated the certs used in these two files and added instructions for how to do so in the resources folder. All of the certs used in the S2A tests expire in 20 yrs. See readme for commands to regenerate.

Done in 19b8f21

ejona86 · 2024-09-16T23:04:07Z

s2a/src/test/resources/client.csr

@@ -0,0 +1,16 @@
+-----BEGIN CERTIFICATE REQUEST-----


It is weird to commit the csr. That is temporary, right?

From the README,the csr is created as part of the steps to regenerate the certs in this folder, which have a lifetime of 20 yrs. This resources directory is similar to the one we have the S2A Go client: https://github.com/google/s2a-go/tree/main/testdata

ejona86 · 2024-09-16T23:05:20Z

s2a/src/test/resources/config.cnf

+subjectAltName = @alt_names
+
+[alt_names]
+IP.1 = ::


What is happening here? How would this ever work?

This is so that the generated certs have localhost IP addr, so that certificate verification passes in tests

ejona86 · 2024-09-16T23:09:09Z

s2a/src/test/java/io/grpc/s2a/handshaker/FakeS2AServerTest.java

+  private static final ImmutableList<ByteString> FAKE_CERT_DER_CHAIN =
+      ImmutableList.of(
+          ByteString.copyFrom(
+              new byte[] {'f', 'a', 'k', 'e', '-', 'd', 'e', 'r', '-', 'c', 'h', 'a', 'i', 'n'}));


FYI: "fake-der-chain".getBytes(StardardCharsets.US_ASCII)

Done in e3f3413

ejona86 · 2024-09-16T23:11:53Z

s2a/src/main/java/io/grpc/s2a/handshaker/tokenmanager/AccessTokenManager.java

+
+/** Manages access tokens for authenticating to the S2A. */
+@ThreadSafe
+public final class AccessTokenManager {


We should move all the subpackages that aren't supposed to be used into an internal subpackage, like io.grpc.s2a.internal.handshaker.tokenmanager here.

ejona86 · 2024-09-16T23:26:04Z

s2a/src/main/java/io/grpc/s2a/handshaker/S2AStub.java

+    @Override
+    public void onNext(SessionResp resp) {
+      verify(!doneReading);
+      responses.offer(Result.createWithResponse(resp));


All the offer()s scare me. The queue is sort of randomly sized (not just 2), which also doesn't feel me with confidence. It'd be much better to use add() which throws an exception if the queue is full instead of silently losing data.

Done in d13bd88

ejona86 · 2024-09-16T23:28:48Z

s2a/src/main/java/io/grpc/s2a/handshaker/S2AStub.java

+        exception = e;
+      }
+      responses.clear();
+      if (exception != null) {


Why do you allow non-exception results here? If there was ever a non-exception present, that guarantees 1) that this code doesn't know what is going on and 2) there is a race (as that message could have been delayed until send() sent the request, at which point it looks like it is the response for the wrong request).

It seems clear to me you've mirrored some of the ALTS code, but I don't know why it was made complicate/worse in ways like this.

This logic existed before we added withWaitForReady on the S2AStub. We had this logic to handle a very specific case:
sometimes the client tries sending the first RPC before the S2A is healthy, so the response returned was always the response to the first RPC. This hasn't caused any problems yet. But now that we are using withWaitForReady, we don't need this type of logic, so I've removed it in: 9efaa6a

ejona86 · 2024-09-16T23:32:08Z

s2a/src/main/java/io/grpc/s2a/handshaker/ProtoUtil.java

+   * @return a {@link String} representing the ciphersuite.
+   * @throws AssertionError if the {@link Ciphersuite} is not one of the supported ciphersuites.
+   */
+  static String convertCiphersuite(Ciphersuite ciphersuite) {


This looks like dead code.

Done in 577b4e5

ejona86 · 2024-09-16T23:33:19Z

s2a/src/main/java/io/grpc/s2a/handshaker/ProtoUtil.java

+   * @return a {@link String} representation of the TLS version.
+   * @throws AssertionError if the {@code tlsVersion} is not one of the supported TLS versions.
+   */
+  static String convertTlsProtocolVersion(TLSVersion tlsVersion) {


@VisibleForTesting? Looks like it is only used privately.

Done in 59573ec

ejona86 · 2024-09-16T23:33:47Z

s2a/src/main/java/io/grpc/s2a/handshaker/ProtoUtil.java

+      case TLS_VERSION_1_0:
+        return "TLSv1";
+      default:
+        throw new AssertionError(


Using an error for an unexpected argument is asking for huge trouble. From what I can tell, this is not currently checked by the caller and this comes from a remote system.

Yes, the tls version info is coming from S2A. Would it be appropriate if we threw an IllegalArgumentException here, and dealt with it in the caller (SslContextFactory)?

ejona86 · 2024-09-17T01:10:56Z

s2a/src/main/java/io/grpc/s2a/handshaker/S2AProtocolNegotiatorFactory.java

+
+      Channel ch = channelPool.getChannel();
+      S2AServiceGrpc.S2AServiceStub stub = S2AServiceGrpc.newStub(ch);
+      S2AStub s2aStub = S2AStub.newInstance(stub);


This Stub is never closed. This causes IntegrationTest (after Larry's reordering of awaitTermination) to not complete tests quickly, as the RPC is outstanding so awaitTermination() hangs. I think once we shut down the top-level channel that releases the S2AHandshakerServiceChannel.ChannelResource after 1 second, which then calls shutdownNow() on the s2a channel which cancels the RPC.

Yes, we never call close on the stub directly. When the
close gets invoked, everything you mentioned happens.

rmehta19 · 2024-09-17T17:50:37Z

Thanks for the comments @ejona86; I'll address them in separate commits at #11534. Let me know if you prefer separate PRs for them, or any other way of organizing.

ejona86 · 2024-09-18T23:27:06Z

There's plenty of smaller/isolated items. Feel free to group those together as you wish, as they are easy to review. A few of the comments might turn into larger/plumbing changes. Some of those may be best to be separate. But overall, if you do lots of the "easy" stuff together that'd be fine to review/merge while you work on something more difficult.

rmehta19 · 2024-09-20T15:46:58Z

@ejona86 , @larry-safran I have addressed all the comments on this PR in the following 3 PRs:

PTAL at these 3 and leave any comments.

I will address the comment to move things to internal package and combine the MTLS and non-MTLS apis in the following 2 PRs, once the above 3 are merged.

Besides these changes, please let me know anything else we can do to enable s2a to be included in a grpc-java release. Thank you!

erm-g assigned erm-g and matthewstevenson88 Apr 18, 2024

rmehta19 mentioned this pull request Apr 18, 2024

gRPC Java S2A protos grpc/grpc-proto#149

Merged

rmehta19 force-pushed the s2a branch 7 times, most recently from e83fee8 to 1991e37 Compare April 20, 2024 00:22

rmehta19 force-pushed the s2a branch 5 times, most recently from 604f9a0 to 3c867c9 Compare April 22, 2024 18:14

matthewstevenson88 reviewed Apr 24, 2024

View reviewed changes

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java Outdated Show resolved Hide resolved

matthewstevenson88 reviewed Apr 24, 2024

View reviewed changes

s2a/src/main/java/io/grpc/s2a/handshaker/S2ATrustManager.java Outdated Show resolved Hide resolved

s2a/src/main/proto/grpc/gcp/common.proto Outdated Show resolved Hide resolved

rmehta19 force-pushed the s2a branch 2 times, most recently from b7b51bc to 73cc37a Compare April 24, 2024 22:22

add s2a java client.

42cd3e8

rmehta19 force-pushed the s2a branch from 73cc37a to 42cd3e8 Compare April 25, 2024 16:49

rmehta19 changed the title ~~[DRAFT]s2a: Add gRPC S2A~~ s2a: Add gRPC S2A Apr 26, 2024

erm-g requested changes Apr 29, 2024

View reviewed changes

ejona86 reviewed Apr 29, 2024

View reviewed changes

s2a/src/test/java/io/grpc/s2a/handshaker/GetAuthenticationMechanismsTest.java Outdated Show resolved Hide resolved

ejona86 reviewed Apr 29, 2024

View reviewed changes

s2a/src/main/java/io/grpc/s2a/S2AChannelCredentials.java Outdated Show resolved Hide resolved

s2a/src/main/proto/grpc/gcp/common.proto Outdated Show resolved Hide resolved

use javax.annotation.Nullable in S2AProtocolNegotiatorFactory.

f96d395

larry-safran reviewed Sep 10, 2024

View reviewed changes

rmehta19 added 4 commits September 12, 2024 09:23

getChannel() doesn't block.

38b0a3a

Remove unnecessary local variable in getAuthMechanism.

12586b1

Invert if statement in ProtoUtil to improve readability.

08f8342

Check localIdentity is null before setting it.

7f26712

larry-safran reviewed Sep 12, 2024

View reviewed changes

s2a/src/test/java/io/grpc/s2a/handshaker/IntegrationTest.java Outdated Show resolved Hide resolved

rmehta19 added 3 commits September 12, 2024 10:32

Check hostname not null or empty.

19583a7

Change channelRead argument ctx to unused.

3198eec

Remove unnecessary waitForReady() in IntegrationTest.

0e059e4

Push down the creation of Optional<S2AIdentity> until S2AProtocolNego…

b021a21

…tiator.

larry-safran reviewed Sep 12, 2024

View reviewed changes

s2a/src/test/java/io/grpc/s2a/handshaker/IntegrationTest.java Show resolved Hide resolved

rmehta19 added 2 commits September 13, 2024 08:31

Wait for servers to be terminated in tearDown in IntegrationTest.java.

752627a

mark unused ctx in channelReadComplete.

a8cacb0

larry-safran approved these changes Sep 13, 2024

View reviewed changes

larry-safran added the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Sep 13, 2024

grpc-kokoro removed the kokoro:run Add this label to a PR to tell Kokoro the code is safe and tests can be run label Sep 13, 2024

larry-safran merged commit b8c1aa5 into grpc:master Sep 14, 2024
15 checks passed

ejona86 reviewed Sep 16, 2024

View reviewed changes

larry-safran mentioned this pull request Sep 16, 2024

s2a: Disabling publishing until it is ready for users #11529

Open

ejona86 reviewed Sep 16, 2024

View reviewed changes

ejona86 reviewed Sep 17, 2024

View reviewed changes

This was referenced Sep 19, 2024

s2a: Address comments on PR#11113 #11534

Merged

s2a,netty: S2AHandshakerServiceChannel doesn't use custom event loop. #11539

Merged

s2a: Address minor comments on PR#11113 #11540

Open

s2a: Add gRPC S2A #11113

s2a: Add gRPC S2A #11113

Conversation

rmehta19 commented Apr 18, 2024 • edited Loading

linux-foundation-easycla bot commented Apr 18, 2024 • edited Loading

matthewstevenson88 commented Apr 22, 2024

rmehta19 commented Apr 22, 2024 • edited Loading

matthewstevenson88 commented Apr 25, 2024

rmehta19 commented Apr 26, 2024

ejona86 left a comment

Choose a reason for hiding this comment

rmehta19 commented Sep 9, 2024

rmehta19 commented Sep 12, 2024

ejona86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmehta19 Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ejona86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmehta19 Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmehta19 commented Sep 17, 2024

ejona86 commented Sep 18, 2024

rmehta19 commented Sep 20, 2024

rmehta19 commented Apr 18, 2024 •

edited

Loading

linux-foundation-easycla bot commented Apr 18, 2024 •

edited

Loading

rmehta19 commented Apr 22, 2024 •

edited

Loading

rmehta19 Sep 19, 2024 •

edited

Loading

rmehta19 Sep 19, 2024 •

edited

Loading