chore(web-modeler): first iteration of Web Modeler troubleshooting gu…

…ide (#2261)

* chore(web-modeler): first iteration of Web Modeler troubleshooting guide

* technical review

* docs(web-modeler): small clarifications for troubleshooting guide

---------

Co-authored-by: Christina Ausley <[email protected]>

docs/components/modeler/desktop-modeler/troubleshooting.md

@@ -101,9 +101,9 @@ Certificate chain
 ...
 ```
 
-## How can I debug log GRPC / Zeebe communication?
+## How can I debug log gRPC / Zeebe communication?
 
-You can also start Desktop Modeler with GRPC logging turned on to get detailed [logging output](#how-to-obtain-desktop-modeler-logs) on communication to Zeebe:
+You can also start Desktop Modeler with gRPC logging turned on to get detailed [logging output](#how-to-obtain-desktop-modeler-logs) on communication to Zeebe:
 
 ```sh
 DEBUG=* ZEEBE_NODE_LOG_LEVEL=DEBUG GRPC_VERBOSITY=DEBUG GRPC_TRACE=all camunda-modeler

docs/self-managed/modeler/web-modeler/configuration/logging.md

@@ -21,6 +21,9 @@ LOGGING_CONFIG=file:/full/path/to/custom-logback-config.xml
 Refer to [Spring Boot's logging documentation](https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.logging.logback)
 for more information on how to customize the logback configuration for specific use cases like logging to a file.
 
+Enabling `DEBUG` logging for the `restapi` component can be useful for troubleshooting purposes, e.g. for
+[debugging Zeebe connection issues](../troubleshooting/troubleshoot-zeebe-connection.md#how-can-i-debug-log-grpc--zeebe-communication).
+
 ## Logging configuration for the `webapp` component
 
 By default, the `webapp` component logs to the Docker container's standard output.
@@ -41,7 +44,7 @@ By default, the `websocket` component logs to the Docker container's standard ou
 
 To enable additional log output to a file, follow these steps:
 
-1. Mount a volume to the directory `/var/www/html/storage/logs`. The logs will be written to a file named `laravel.log`located inside this directory.
+1. Mount a volume to the directory `/var/www/html/storage/logs`. The logs will be written to a file named `laravel.log` located inside this directory.
 2. Adjust the following environment variable:
    ```
    LOG_CHANNEL=single

docs/self-managed/modeler/web-modeler/troubleshooting/troubleshoot-database-connection.md

@@ -0,0 +1,40 @@
+---
+id: troubleshoot-database-connection
+title: "Troubleshooting database connection issues"
+sidebar_label: "Database connection"
+---
+
+:::note
+Web Modeler Self-Managed is available to [enterprise customers](../../../../reference/licenses.md#web-modeler) only.
+:::
+
+You try to start Web Modeler, and encounter issues with the database connection.
+
+## Secure connection to standard PostgreSQL
+
+Refer to the [database configuration guide](../configuration/database.md#configuring-ssl-for-the-database-connection)
+for details on how to configure a secure connection to PostgreSQL.
+
+## Secure connection to Amazon Aurora fails
+
+You configured a custom SSL certificate in your remote Amazon Aurora PostgreSQL instance and want Web Modeler to accept
+that certificate.
+
+### Add Amazon Root CA to trust store
+
+By default, the Java version used by `modeler-restapi` ships with the Amazon Root CA.
+
+If you passed a custom trust store to `modeler-restapi`'s JVM process (e.g. via `JAVA_TOOL_OPTIONS` as described in
+[the Zeebe connection troubleshooting guide](./troubleshoot-zeebe-connection.md#provide-the-certificate-to-the-jvm-trust-store)),
+ensure the Amazon Trust Services CA are in `modeler-restapi`'s trust store (see the
+[Amazon Aurora documentation](https://aws.amazon.com/blogs/security/how-to-prepare-for-aws-move-to-its-own-certificate-authority/)).
+
+## IAM authentication against Amazon Aurora fails
+
+You switched from standard username/password authentication to IAM authentication and Web Modeler can't obtain a connection to the database.
+
+### Ensure the IAM account has all privileges to the Web Modeler database
+
+After switching from standard username/password authentication to IAM authentication, privileges to Web Modeler's
+database might still be associated with the old username.
+Ensure the IAM account has all privileges to the Web Modeler database.

docs/self-managed/modeler/web-modeler/troubleshooting/troubleshoot-zeebe-connection.md

@@ -0,0 +1,146 @@
+---
+id: troubleshoot-zeebe-connection
+title: "Troubleshooting Zeebe connection issues"
+sidebar_label: "Zeebe connection"
+---
+
+:::note
+Web Modeler Self-Managed is available to [enterprise customers](../../../../reference/licenses.md#web-modeler) only.
+:::
+
+You try to connect (i.e., to deploy) to a remote Zeebe cluster and Web Modeler tells you:
+
+- The entered cluster endpoint URL should "point to a running Zeebe cluster."
+- An unknown error occurred and to "check Zeebe cluster status."
+
+To resolve this issue, check if you can connect to Zeebe through another client, i.e.,
+[`zbctl`](../../../../apis-tools/cli-client/index.md).
+If that doesn't work, resolve the general connection issue first (see [the platform deployment
+troubleshooting section](https://docs.camunda.io/docs/self-managed/platform-deployment/troubleshooting/), for example.)
+
+If that works, further debug your Zeebe connection with the help of the information stated below. Enabling [debug logging in `modeler-restapi`](#how-can-i-debug-log-grpc--zeebe-communication) may also help to
+understand the issue.
+
+## Secure connection to Zeebe fails
+
+If you provide a cluster endpoint URL starting with `https`, Web Modeler will try to establish a secure connection to
+the Zeebe instance.
+In the process, it strictly validates the server's Application-Layer Protocol Negotiation (ALPN) support and its certificates
+presented against well-known certificate authorities.
+Failure to connect may have several reasons:
+
+### Configure the endpoint to accept secure connections
+
+Ensure you properly configure the remote endpoint to accept secure connections.
+Refer to the [Zeebe Gateway configuration documentation](../../../zeebe-deployment/security/secure-client-communication.md#gateway)
+for additional information.
+
+### Configure the endpoint to support ALPN
+
+[Inspect the connection](#how-can-i-get-details-about-a-secure-remote-connection) to understand if ALPN is supported
+by the server.
+
+Secure connections to Zeebe require an Ingress controller that supports HTTP/2 over TLS with protocol negotiation via ALPN.
+Ensure you properly [configured your Zeebe ingress to support ALPN](../../../platform-deployment/troubleshooting.md#zeebe-ingress-grpc).
+
+### Configure `modeler-restapi` to trust a custom Zeebe SSL certificate
+
+[Inspect the connection](#how-can-i-get-details-about-a-secure-remote-connection) to understand which certificates are
+being returned by the server and ensure you configure Web Modeler for [custom SSL certificates](#how-can-i-provide-a-custom-zeebe-ssl-certificate).
+
+If intermediate signing authorities sign the server certificate, ensure the remote endpoint [serves both server and
+intermediate certificates](https://nginx.org/en/docs/http/configuring_https_servers.html#chains) to Web Modeler.
+
+### Make the OAuth token cache location writeable for the `modeler-restapi` process
+
+When using the `OAuth` authentication method for deploying to Zeebe, Web Modeler caches OAuth tokens in a file-based
+cache.
+By default, the cache location is writeable by the `modeler-restapi` process.
+If you run `modeler-restapi` as a non-root user (e.g. via Kubernetes' `securityContext.runAsUser` option),
+you must ensure to provide a writeable cache file location to `modeler-restapi` via the `ZEEBE_CLIENT_CONFIG_PATH`
+environment variable:
+
+```
+ZEEBE_CLIENT_CONFIG_PATH=/path/to/credentials/cache.txt
+```
+
+## How can I provide a custom Zeebe SSL certificate?
+
+You configured a custom SSL certificate in your (remote) Zeebe endpoint and want Web Modeler to accept that certificate.
+Web Modeler strictly validates the remote server certificate trust chain.
+If you use a custom SSL server certificate, you must make the signing CA certificate known to Web Modeler, not the
+server certificate itself.
+
+### Provide the certificate via an environment variable
+
+`modeler-restapi` reads a trusted certificate from the environment variable `ZEEBE_CA_CERTIFICATE_PATH`.
+This solution is recommended for most users:
+
+```sh
+ZEEBE_CA_CERTIFICATE_PATH=/path/to/certificate
+```
+
+The provided path has to be accessible from the `modeler-restapi` container (e.g. via a mounted volume).
+
+### Provide the certificate to the JVM trust store
+
+Alternatively, you may pass a custom trust store to `modeler-restapi` via the environment variable `JAVA_TOOL_OPTIONS`:
+
+```sh
+JAVA_TOOL_OPTIONS="-Djavax.net.ssl.trustStore=/path/to/truststore.jks -Djavax.net.ssl.trustStorePassword=changeit"
+```
+
+Analogous to above, the provided path has to be accessible from the `modeler-restapi` container (e.g. via a mounted volume).
+
+:::caution
+Be aware that passing a custom trust store via `JAVA_TOOL_OPTIONS` overrides the default JVM trust store.
+This means that all certificates shipped by default with the Java runtime (e.g. the Amazon Trust Services CA for
+connecting to a secured Aurora instance) are no longer trusted unless explicitly added.
+:::
+
+## How can I get details about a secure remote connection?
+
+You can use the following command to retrieve information about HTTP/2 over TLS support (ALPN) and certificates provided
+by a remote endpoint:
+
+```sh
+> openssl s_client -alpn h2 -connect google.com:443 -servername google.com
+[...]
+---
+Certificate chain
+ 0 s:/CN=*.google.com
+   i:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
+ 1 s:/C=US/O=Google Trust Services LLC/CN=GTS CA 1C3
+   i:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
+ 2 s:/C=US/O=Google Trust Services LLC/CN=GTS Root R1
+   i:/C=BE/O=GlobalSign nv-sa/OU=Root CA/CN=GlobalSign Root CA
+---
+[...]
+---
+New, TLSv1/SSLv3, Cipher is AEAD-CHACHA20-POLY1305-SHA256
+Server public key is 256 bit
+Secure Renegotiation IS NOT supported
+Compression: NONE
+Expansion: NONE
+ALPN protocol: h2
+SSL-Session:
+    Protocol  : TLSv1.3
+    Cipher    : AEAD-CHACHA20-POLY1305-SHA256
+    Session-ID:
+    Session-ID-ctx:
+    Master-Key:
+    Start Time: 1687516295
+    Timeout   : 7200 (sec)
+    Verify return code: 0 (ok)
+---
+```
+
+## How can I debug log gRPC / Zeebe communication?
+
+You can also start `modeler-restapi` with gRPC debug logging turned on to get detailed [logging
+output](../configuration/logging.md) on communication to Zeebe:
+
+```sh
+LOGGING_LEVEL_IO_GRPC=TRACE
+LOGGING_LEVEL_IO_CAMUNDA_MODELER=DEBUG
+```

docs/self-managed/platform-deployment/troubleshooting.md

@@ -37,7 +37,7 @@ global:
 Zeebe requires an Ingress controller that supports `gRPC` which is built on top of `HTTP/2` transport layer. Therefore, to expose Zeebe-Gateway externally, you need the following:
 
 1. An Ingress controller that supports `gRPC` ([Ingress-NGINX controller](https://github.com/kubernetes/ingress-nginx) supports it out of the box).
-2. TLS enabled (HTTPS) in the Zeebe-Gateway Ingress object.
+2. TLS (HTTPS) via [Application-Layer Protocol Negotiation (ALPN)](https://www.rfc-editor.org/rfc/rfc7301.html) enabled in the Zeebe-Gateway Ingress object.
 
 However, according to the official Kubernetes documentation about [Ingress TLS](https://kubernetes.io/docs/concepts/services-networking/ingress/#tls):
 

sidebars.js

@@ -1184,6 +1184,10 @@ module.exports = {
                 "self-managed/modeler/web-modeler/configuration/database",
                 "self-managed/modeler/web-modeler/configuration/logging",
               ],
+              Troubleshooting: [
+                "self-managed/modeler/web-modeler/troubleshooting/troubleshoot-database-connection",
+                "self-managed/modeler/web-modeler/troubleshooting/troubleshoot-zeebe-connection",
+              ],
             },
           ],
         },

versioned_docs/version-8.2/components/modeler/desktop-modeler/troubleshooting.md

@@ -101,9 +101,9 @@ Certificate chain
 ...
 ```
 
-## How can I debug log GRPC / Zeebe communication?
+## How can I debug log gRPC / Zeebe communication?
 
-You can also start Desktop Modeler with GRPC logging turned on to get detailed [logging output](#how-to-obtain-desktop-modeler-logs) on communication to Zeebe:
+You can also start Desktop Modeler with gRPC logging turned on to get detailed [logging output](#how-to-obtain-desktop-modeler-logs) on communication to Zeebe:
 
 ```sh
 DEBUG=* ZEEBE_NODE_LOG_LEVEL=DEBUG GRPC_VERBOSITY=DEBUG GRPC_TRACE=all camunda-modeler

versioned_docs/version-8.2/self-managed/modeler/web-modeler/configuration/logging.md

@@ -18,16 +18,19 @@ custom configuration file and let the `restapi` know of it by specifying the fol
 LOGGING_CONFIG=file:/full/path/to/custom-logback-config.xml
 ```
 
-Please refer to the [Spring Boot's logging documentation](https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.logging.logback)
+Refer to [Spring Boot's logging documentation](https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.logging.logback)
 for more information on how to customize the logback configuration for specific use cases like logging to a file.
 
+Enabling `DEBUG` logging for the `restapi` component can be useful for troubleshooting purposes, e.g. for
+[debugging Zeebe connection issues](../troubleshooting/troubleshoot-zeebe-connection.md#how-can-i-debug-log-grpc--zeebe-communication).
+
 ## Logging configuration for the `webapp` component
 
 By default, the `webapp` component logs to the Docker container's standard output.
 
 ### Logging to a file
 
-If you want to enable additional log output to a file, adjust the following environment variable:
+To enable additional log output to a file, adjust the following environment variable:
 
 ```
 LOG_FILE_PATH=/full/path/to/log/file.log
@@ -39,13 +42,10 @@ By default, the `websocket` component logs to the Docker container's standard ou
 
 ### Logging to a file
 
-If you want to enable additional log output to a file, follow these steps:
-
-1. Mount a volume to the directory `/var/www/html/storage/logs`. The logs will be written to a file named `laravel.log`
-   located inside this directory.
+To enable additional log output to a file, follow these steps:
 
+1. Mount a volume to the directory `/var/www/html/storage/logs`. The logs will be written to a file named `laravel.log` located inside this directory.
 2. Adjust the following environment variable:
-
    ```
    LOG_CHANNEL=single
    ```

versioned_docs/version-8.2/self-managed/modeler/web-modeler/troubleshooting/troubleshoot-database-connection.md

@@ -0,0 +1,40 @@
+---
+id: troubleshoot-database-connection
+title: "Troubleshooting database connection issues"
+sidebar_label: "Database connection"
+---
+
+:::note
+Web Modeler Self-Managed is available to [enterprise customers](../../../../reference/licenses.md#web-modeler) only.
+:::
+
+You try to start Web Modeler, and encounter issues with the database connection.
+
+## Secure connection to standard PostgreSQL
+
+Refer to the [database configuration guide](../configuration/database.md#configuring-ssl-for-the-database-connection)
+for details on how to configure a secure connection to PostgreSQL.
+
+## Secure connection to Amazon Aurora fails
+
+You configured a custom SSL certificate in your remote Amazon Aurora PostgreSQL instance and want Web Modeler to accept
+that certificate.
+
+### Add Amazon Root CA to trust store
+
+By default, the Java version used by `modeler-restapi` ships with the Amazon Root CA.
+
+If you passed a custom trust store to `modeler-restapi`'s JVM process (e.g. via `JAVA_TOOL_OPTIONS` as described in
+[the Zeebe connection troubleshooting guide](./troubleshoot-zeebe-connection.md#provide-the-certificate-to-the-jvm-trust-store)),
+ensure the Amazon Trust Services CA are in `modeler-restapi`'s trust store (see the
+[Amazon Aurora documentation](https://aws.amazon.com/blogs/security/how-to-prepare-for-aws-move-to-its-own-certificate-authority/)).
+
+## IAM authentication against Amazon Aurora fails
+
+You switched from standard username/password authentication to IAM authentication and Web Modeler can't obtain a connection to the database.
+
+### Ensure the IAM account has all privileges to the Web Modeler database
+
+After switching from standard username/password authentication to IAM authentication, privileges to Web Modeler's
+database might still be associated with the old username.
+Ensure the IAM account has all privileges to the Web Modeler database.