Components failing to get adresses for interface

[nesspll]

Contex: EKS IPv6

values config file

global:
  reportingEnabled: false

serviceAccount:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::258570154***:role/pyroscope-storage



pyroscope:
  replicaCount: 1
  cluster_domain: .cluster.local

  extraEnvVars:
    POD_IP:
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

  extraArgs:
    memberlist.bind-addr: "$(POD_IP)"
    log.level: "debug"
    ingester.enable-inet6: "true"




  storage:
    backend: s3
    s3:
      endpoint: s3.us-east-1.amazonaws.com
      region: us-east-1
      bucket_name: pyroscope-blocks

  memberlist:
      join_members:
        - pyroscope-gossip-ring.pyroscope-new.svc.cluster.local:7946
      bind_addr: "[::]"
      bind_port: 7946



  image:
    repository: hub.docker.com/grafana/pyroscope
    pullPolicy: IfNotPresent  

  serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::258570154***:role/pyroscope-storage  

  server:
    http_listen_network: "tcp"
    http_listen_address: "[::]"
    http_listen_port: 4040
    grpc_listen_network: "tcp"
    grpc_listen_address: "[::]"
    grpc_listen_port: 9095

  components:
    querier:
      kind: Deployment
      replicaCount: 3
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 256Mi
          cpu: 1
      storage:
        backend: s3
        s3:
          endpoint: s3.us-east-1.amazonaws.com
          region: us-east-1
          bucket_name: pyroscope-blocks



    distributor:
      kind: Deployment
      replicaCount: 2
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 256Mi
          cpu: 500m
      storage:
        backend: s3
        s3:
          endpoint: s3.us-east-1.amazonaws.com
          region: us-east-1
          bucket_name: pyroscope-blocks
      lifecycler:
        ring:
          kvstore:
            store: memberlist
      extraEnv:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      tokens_file_path: /var/pyrscope/tokens/distributor-tokens.json             

    ingester:
      kind: StatefulSet
      replicaCount: 3
      terminationGracePeriodSeconds: 600
      storage:
        backend: s3
        s3:
          endpoint: s3.us-east-1.amazonaws.com
          region: us-east-1
          bucket_name: pyroscope-blocks
      resources:
        limits:
          memory: 4Gi
        requests:
          memory: 2Gi
          cpu: 1
      extraEnv:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      tokens_file_path: /var/pyrscope/tokens/ingester-tokens.json
      lifecycler:
        address: "${POD_IP}"
        enable_inet6: true
        interface_names:
          - eth0
          - lo
          - v4if0
        ring:
          kvstore:
            store: memberlist

    compactor:
      kind: StatefulSet
      replicaCount: 3
      terminationGracePeriodSeconds: 1200
      storage:
        backend: s3
        s3:
          endpoint: s3.us-east-1.amazonaws.com
          region: us-east-1
          bucket_name: pyroscope-blocks
      persistence:
        enabled: false
      resources:
        limits:
          memory: 4Gi
        requests:
          memory: 2Gi
          cpu: 1
      lifecycler:
        ring:
          kvstore:
            store: memberlist
      extraEnv:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      tokens_file_path: /var/pyrscope/tokens/compactor-tokens.json
          
analytics:
    reporting_enabled: false

minio:
  enabled: true
  image:  
    repository: quay.io/minio/minio
  mcImage:
    repository: quay.io/minio/mc

alloy: 
  enabled:  false

---

Error in Ingester component:

caller=netutil.go:85 level=debug msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false
caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr="invalid IP" inf=eth0
caller=netutil.go:104 level=debug msg="ignoring address" ipAddr="invalid IP" inf=eth0
caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"
caller=netutil.go:118 level=debug msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"
failed running pyroscope: error initialising module: version: no useable address found for interfaces [eth0 en0]

ts=2024-11-28T21:36:48.448324076Z caller=server.go:353 level=info msg="server listening on addresses" http=[::]:4040 grpc=[::]:9095ts=2024-11-28T21:36:48.449615341Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/admin auth=false gzip=truets=2024-11-28T21:36:48.451322992Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/api/swagger.json auth=false gzip=truets=2024-11-28T21:36:48.451370724Z caller=api.go:106 level=debug msg="api: registering route" methods=GET,POST,PUT,DELETE,HEAD,OPTIONS prefix=/api auth=false gzip=truets=2024-11-28T21:36:48.451385186Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/debug/fgprof auth=false gzip=truets=2024-11-28T21:36:48.451404633Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/static/ auth=false gzip=truets=2024-11-28T21:36:48.45141978Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/ui auth=false gzip=truets=2024-11-28T21:36:48.451433401Z caller=api.go:106 level=debug msg="api: registering route" methods=GET prefix=/assets/ auth=false gzip=truets=2024-11-28T21:36:48.451777831Z caller=api.go:101 level=debug msg="api: registering route" methods=GET path=/memberlist auth=false gzip=truets=2024-11-28T21:36:48.452556139Z caller=netutil.go:85 level=debug msg="looking for addresses" inf="[eth0 en0]" inet6enabled=falsets=2024-11-28T21:36:48.452654829Z caller=netutil.go:102 level=debug msg="detected highest quality address" ipAddr="invalid IP" inf=eth0ts=2024-11-28T21:36:48.452671394Z caller=netutil.go:104 level=debug msg="ignoring address" ipAddr="invalid IP" inf=eth0ts=2024-11-28T21:36:48.45270517Z caller=netutil.go:90 level=warn msg="error getting addresses for interface" inf=en0 err="route ip+net: no such network interface"ts=2024-11-28T21:36:48.452723232Z caller=netutil.go:118 level=debug msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"failed running pyroscope: error initialising module: version: no useable address found for interfaces [eth0 en0]

[kolesnikovae]

Hi @nesspll!

I'm afraid there isn't a single option to enable IPv6. The following flags or corresponding options in the configuration need to be set:

• -ingester.enable-inet6
• -distributor.ring.instance-enable-ipv6
• -store-gateway.sharding-ring.instance-enable-ipv6
• -compactor.ring.instance-enable-ipv6
• -overrides-exporter.ring.instance-enable-ipv6
• -query-scheduler.ring.instance-enable-ipv6

I also notices some inconsistencies in the values.yaml file you provided – it includes some settings that should be set in the Pyroscope's configuration (pyroscope.config) or passed as extra flags. Please refer to the chart docs for more details

[nesspll]

The flags, should the be set under the section of pyroscope.extraArgs also can you point as an example to those inconsistencies, that should move to pyroscope.config section

[ipversiongent]

Seeing the following error for the distributor component, even after adding in the code below (we tried multiple times/places to add the ipv6 code, seems like the 'interface' doesn't recognize ipv6?"

`ts=2024-11-29T11:16:59.44059492Z caller=netutil.go:118 level=debug component=distributor msg="detected IP address after looking up all configured interface names" ipAddr="invalid IP"

failed running pyroscope: error initialising module: distributor: failed to build distributors' lifecycler config: no useable address found for interfaces [eth0 en0]

ts=2024-11-29T11:16:59.440431689Z caller=netutil.go:85 level=debug component=distributor msg="looking for addresses" inf="[eth0 en0]" inet6enabled=false

s=2024-11-29T11:16:59.440276625Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
`

###code

distributor:   
      kind: Deployment
      replicaCount: 2
      resources:
        limits:
          memory: 1Gi
        requests:
          memory: 256Mi
          cpu: 500m
      storage:
        backend: s3
        s3:
          endpoint: s3.us-east-1.amazonaws.com
          region: us-east-1
          bucket_name: bt-sre-poc-pyroscope-blocks
      ring:
        kvstore:
          store: memberlist
        instance_enable_ipv6: true 
        instance_addr: $(POD_IP) 
        instance_port: 9095
        instance_interface_names:
          - eth0
      extraEnv:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
      tokens_file_path: /var/pyrscope/tokens/distributor-tokens.json

Surprisingly, the memberlist advertisement works. Also, we have managed to successfully deploy the compactors!

[ipversiongent]

@kolesnikovae

Does this mean only IPV4 is supported (and not Ipv6) and it will throw an error regardless of IPv6 assignment?

https://github.com/grafana/pyroscope/blob/main/pkg/util/net.go#L16
// GetFirstAddressOf returns the first IPv4 address of the supplied interface names, omitting any 169.254.x.x automatic private IPs if possible.

The pods themsevles seem to be getting IPv6 addresses.

pyroscope-ingester-0 0/1 CrashLoopBackOff 6 (4m59s ago) 11m 2600:1f18:5562:ae03:33d2::8 ip-10-48-4-6.ec2.internal <none> <none> pyroscope-ingester-1 0/1 CrashLoopBackOff 6 (4m34s ago) 10m 2600:1f18:5562:ae04:7a32::3 ip-10-48-4-98.ec2.internal <none> <none> pyroscope-ingester-2 0/1 CrashLoopBackOff 6 (4m47s ago) 10m 2600:1f18:5562:ae03:93b5::b ip-10-48-4-58.ec2.internal <none>

[ipversiongent]

@cyriltovena , any chance you have an input on this one?

We're trying to demo this to our executive leadership and it would be great to have an answer on this one.

Appreciate it.

[kolesnikovae]

@ipversiongent you're right about GetFirstAddressOf. I believe there are several issues in Pyroscope that currently make using IPv6 impossible.

Let's aim to fix this by the next release. This is the first patch: #3731. I suspect other issues may arise once IPv6 is enabled. It would be incredibly helpful if you could test the patch and let us know if it resolves the issue for you.

[ipversiongent]

@kolesnikovae ,

Since we're running a sandbox deployment, we will try to provide feedback on our IPv6 deployment. Is the expecation to have the changes deployed in the new release v1.11?

Let's stay in touch and maybe address this issue methodically. Thank you for all your support and fast responses. Appreciate all your work.

[kolesnikovae]

Awesome! Thanks for your help!

We'll discuss this internally and carry out some integration testing. After that, if everything goes well, the fix will be included in our weekly build next Monday, available publicly (kind of release candidate). I'll reach out to you once everything is ready. Thanks again!

[ipversiongent]

@kolesnikovae ,

Hello Anton. On another note, unrelated to this, is there any place we can find deployment scenarios alongside instrumentation methods? I looked at Rob Chapman's book but the section on Pyroscope is fairly limited and purely theoretical - without any details on deployment and integration strategies.
Apologies for reaching out in this issue and possibly convoluting the conversation. Thank you again, and best of luck. Regards.

[kolesnikovae]

Hello @ipversiongent! We've released the f96 weekly build with the fix included – I hope this will address the problem you encountered.

is there any place we can find deployment scenarios alongside instrumentation methods? I looked at Rob Chapman's book but the section on Pyroscope is fairly limited and purely theoretical - without any details on deployment and integration strategies.

I apologize for the delay; I just noticed the comment.

As for the Pyroscope deployment scenarios: we run it in k8s ourselves and therefore I'd recommend using the helm chart. Depending on the availability and capacity requirements, you may prefer the microservice deployment (designed for HA and horizontal scalability) or start with a single-binary deployment and migrate later. If you have specific questions, I'd be happy to address them.

The client side requires more thorough analysis, though. I recommend starting from this page. In short, there are 3 ways:

1. Instrument your applications with language-specific SDKs: gives you the full control over the profiling process and offers more capabilities tailored to the application language/runtime.
2. Auto-instrumentation via Alloy:
   • Go: profiles are pulled from the discovered targets via the http endpoint by Alloy (like Prometheus metrics). A perfect choice if you run Go applications in k8s/docker/VMs/bare-metal.
   • Java: Alloy may be configured to discover Java processes and attach the profiler automatically. Good choice if you run Java applications in k8s/docker/VMs/bare-metal.
   • eBPF: offers the quickest way to collect CPU profiles for compiled languages. Might not be ideal for interpreted languages like Python or Ruby.
3. Auto-instrumentation via eBPF OpenTelemetry collector. Same as Alloy eBPF but integrated with OTel ecosystem. Experimental: many functions may be incomplete or not work as intended.