[BUG] Jaeger Hotrod demo does not work

[sergey-rsa]

Describe the bug
data-prepper/examples/jaeger-hotrod does not work
no traces in Kibana
and error in logs
2023/02/14 12:00:26 Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused

To Reproduce

1. clone repo
2. docker-compose up --build
3. open application in browser, press any button
4. See error

Screenshots

Environment (please complete the following information):

• OS: [linux Mint 21.1]

[sshivanii]

Hi @derangeer,

Thanks for reporting this. I was able to reproduce the same error when I ran the jaeger-hotrod example as well. We will add this Bug to our tracking board and this issue will be updated with any further updates.

jaeger-hotrod-jaeger-hot-rod-1  | 2023/02/21 18:10:03 Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused
jaeger-hotrod-jaeger-hot-rod-1  | 2023/02/21 18:10:03 Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused
jaeger-hotrod-jaeger-hot-rod-1  | 2023/02/21 18:10:03 Post "http://localhost:14268/api/traces": dial tcp 127.0.0.1:14268: connect: connection refused

[rizwan-balouch]

i am also facing same, any solution ?

[fridgepoet]

In the docker-compose you could change the image:jaegertracing/example-hotrod ‘s :latest to :1.41.0 which was working for me

[omishagupta]

As mentioned by @fridgepoet, updating the docker image to 1.41.0 worked like charm.

[beachwood23]

I am still seeing this issue, although with a slightly different error message. I see a repeating of:

data-prepper                    | 2023-11-06T18:30:05,749 [service-map-pipeline-sink-worker-4-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
data-prepper                    | See https://opensearch.org/docs/latest/clients/java-rest-high-level/ for troubleshooting. 
data-prepper                    | 2023-11-06T18:30:05,831 [raw-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [raw-pipeline] - sink is not ready for execution, retrying
data-prepper                    | 2023-11-06T18:30:05,831 [raw-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Initializing OpenSearch sink
data-prepper                    | 2023-11-06T18:30:05,831 [raw-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the username provided in the config.
data-prepper                    | 2023-11-06T18:30:05,831 [raw-pipeline-sink-worker-6-thread-1] INFO  org.opensearch.dataprepper.plugins.sink.opensearch.ConnectionConfiguration - Using the cert provided in the config.
data-prepper                    | 2023-11-06T18:30:05,847 [raw-pipeline-sink-worker-6-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

[t1]

@beachwood23 : that looks like your server uses a certificate that is either self-signed or signed by a CA that is not in your keychain.

[beachwood23]

Thanks @t1 , much appreciated. I am not defining any certificate myself. I believe the example is defining its own certificate from the examples folder here:

data-prepper/examples/jaeger-hotrod/docker-compose.yml

Line 25 in 2d7e4d3

- ../demo/demo-data-prepper.crt:/etc/demo-data-prepper.crt

Is it possible that this demo-data-prepper.crt cert is expired? I see that it was last updated 3 years ago: https://github.com/opensearch-project/data-prepper/blob/main/examples/demo/demo-data-prepper.crt

Or is there some other setup step that I can add to the README?

Cheers.

Update:

I see that the certificate should still be valid, for another 7 years.

$ openssl x509 -in demo-data-prepper.crt -text -noout 
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            c1:60:9c:6e:bb:3b:1b:73
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = data-prepper
        Validity
            Not Before: Nov 30 17:44:38 2020 GMT
            Not After : Nov 28 17:44:38 2030 GMT
        Subject: CN = data-prepper
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c1:b9:24:c8:de:8c:fd:64:94:62:55:00:9d:4d:
                    f6:d0:94:6f:53:58:25:f0:b8:88:e3:25:c5:70:be:
                    05:63:84:8b:a3:e5:52:c4:39:65:29:0b:c9:eb:6d:
                    ee:3f:de:db:46:2e:ea:8e:05:58:8b:97:0d:9a:7a:
                    53:7b:64:03:60:17:2d:f7:38:d1:7b:90:38:78:c4:
                    53:29:e2:b6:5d:14:ac:42:3a:c9:94:a5:b4:83:ee:
                    14:5f:f2:65:51:16:fd:f4:6f:1f:7f:fd:b5:63:2c:
                    ff:2a:8b:64:0a:b8:77:3c:58:e7:af:32:c7:4d:10:
                    d1:48:e8:18:22:23:9d:84:a1:53:f2:60:92:08:31:
                    a1:36:ae:2a:46:aa:44:15:9f:4e:13:93:ea:d6:cf:
                    7c:d0:f8:dc:8e:70:4c:68:c3:df:3e:a7:13:08:8c:
                    06:a7:31:41:0b:4e:93:23:40:25:aa:d8:7b:55:da:
                    79:ea:71:fa:aa:34:26:9e:8a:0e:cd:48:34:6f:42:
                    d3:cb:f3:81:63:cb:bc:b8:07:5a:52:e7:9f:21:06:
                    67:ce:c9:97:06:1e:8f:e0:c4:bc:50:9a:f6:f5:50:
                    ac:b8:6d:c2:fb:29:df:07:dd:a3:6b:2d:9f:33:b2:
                    bf:a1:7b:77:0c:f5:ae:d2:65:b8:a2:94:02:dd:72:
                    fd:0d
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        ab:f0:7c:b8:9f:95:7d:16:77:9e:04:c6:c4:2f:2e:fc:19:6e:
        b5:54:d3:3d:75:d2:1d:8d:f8:4e:d2:5e:df:4a:e0:d8:47:b0:
        03:6c:5f:69:96:c5:ba:94:ff:46:0c:61:0a:68:7c:f3:b4:5c:
        94:aa:9a:8d:bc:60:3f:de:c8:a1:1a:cc:98:3b:b4:52:12:12:
        a4:39:81:f5:99:33:c0:64:7c:77:23:6c:73:22:c6:d4:43:9b:
        a4:a9:37:37:de:41:3f:7b:0b:4c:d9:a3:99:00:0f:87:74:63:
        71:70:56:34:39:16:4b:e5:98:ce:ab:09:a2:74:37:2b:d9:05:
        42:6c:78:2b:16:a5:9b:15:da:f1:74:41:b3:10:71:0d:25:c6:
        3a:98:57:1b:3c:0a:7c:29:f5:b7:c5:ed:93:f7:8b:92:83:e4:
        9c:6d:56:6f:34:83:a3:08:41:da:18:0d:0a:ad:32:38:59:f2:
        b0:a0:e5:ed:52:e8:36:cf:9c:b4:65:99:79:33:e1:c1:5f:0f:
        85:7f:ef:e9:5d:4a:91:2d:df:c4:01:b4:21:56:6a:57:de:9b:
        ee:1d:b5:b1:17:f0:5a:13:72:23:59:7c:29:1e:52:8b:69:3b:
        68:cf:4b:52:33:ce:08:40:cb:2e:9b:2c:97:76:30:79:7c:02:
        71:91:75:08

Still looking into why the provided certificate is causing issues.

[beachwood23]

I haven't been able to debug the certificates issue. I am not sure which part is going wrong there.

I have tried to set this up without using any security at all, and am seeing a different failed port connection:

jaeger-hotrod-jaeger-hot-rod-1  | 2023/11/07 22:38:07 traces export: Post "http://localhost:4318/v1/traces": dial tcp 127.0.0.1:4318: connect: connection refused

[fridgepoet]

I started getting the following recently:
WARN org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink - Failed to initialize OpenSearch sink, retrying: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

What worked is for me is replacing image: opensearchproject/opensearch's :latest to :2.9.0

The other replacement still seems necessary, too: #2273 (comment)

(@beachwood23 I suspect this was your problem)

[frodopwns]

I am getting this connection refused error when trying to use the demo app as well.

[linghengqian]

• I noticed this issue while investigating [BUG] Docker Image's I18N_LOCALE environment variable does not seem to work OpenSearch-Dashboards#8075 . I was surprised that the contents of root-ca.pem were actually mounted to a directory on the host, which limited user options and caused a lot of confusion. I've opened Fixes Jaeger Hotrod demo failure on OpenSearch 2.16.0 #4921 to fix this issue.

[linghengqian]

In the docker-compose you could change the image:jaegertracing/example-hotrod ‘s :latest to :1.41.0 which was working for me

• I've opened Removes Jaeger Agent from Jaeger Hotrod Demo #4923 to discuss this.