In Monolithic cluster mode, is GRPC port really used externally ?

[wally007]

I've read some posts on this forum that GRPC port must be enabled and externally available even if using monolithic mode with 3 instances.

I'm running RHEL9.3 with firewall enabled and only opened port 9096 for http server and 7946 for GOSSIP protocol.
When I start the cluster, I see all 3 nodes and cluster as healthy. Also all modules like compactor see all 3 instances in the hashring and as healthy.

I'm using the following config (the config is deployed by Ansible, so bind address 192.168.123.x and hostname are approproately changed on each host)

target: all

usage_stats:
  enabled: false

multitenancy_enabled: false

server:
  log_level: info
  http_listen_address: mimir3.ref
  http_listen_port: 9096
  http_tls_config:
    cert_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    key_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    client_ca_file: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    client_auth_type: RequireAndVerifyClientCert
  grpc_listen_address: '127.0.0.1'
  grpc_listen_port: 9095

common:
  storage:
    backend: s3
    s3:
      endpoint: xxx.xxx:443
      access_key_id: xxxx
      secret_access_key: xxx
      insecure: false
      bucket_name: grafana-mimir

## Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  storage_prefix: blocks
  bucket_store:
    sync_interval: 5m
    ignore_blocks_within: 0s
    metadata_cache:
      bucket_index_content_ttl: 1m
      tenants_list_ttl: 1m
      tenant_blocks_list_ttl: 1m
      metafile_doesnt_exist_ttl: 1m
  tsdb:
    flush_blocks_on_shutdown: true

querier:
  lookback_delta: 5m ## after this period, series is considered stale (default 5m in prometheus)

limits:
  max_global_series_per_user: 0 ## disable default per user limits
  out_of_order_time_window: 30m ## Non-zero value enables out-of-order samples
  compactor_block_upload_enabled: true
  compactor_blocks_retention_period: 1y ## Delete from storage metrics data older than configured

compactor:
  cleanup_interval: 5m

## Cluster settings
memberlist:
  randomize_node_name: false
  bind_addr: ['192.168.123.183']
  bind_port: 7946
  join_members: [mimir2,mimir3,mimir1]
  tls_enabled: true
  tls_server_name: mimir.ref
  tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
  tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
  tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
  tls_insecure_skip_verify: false
  cluster_label: mimir_group

[dimitarvdimitrov]

the gRPC port would end up being used when running queries and for completing remote-write requests. The cluster might appear healthy before that, but reads and writes won't be able to work without the gRPC port (9095 in your case & the default) is accessible between instances.

[wally007]

Thanks for heads up. I have not written or read from the cluster yet :-)

I went through the documentation and enabled TLS on all GRPC modules, however the GPRC listen address does not seem to be taken from the congfig when running in monolithic mode.
I get the following error:

Jan 04 12:04:31 mimir3 mimir[160989]: ts=2024-01-04T12:04:31.635857596Z caller=frontend_processor.go:82 level=error msg="error contacting frontend" address=127.0.0.1:9095 err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:9095: connect: connection refused\""

I'm not sure why its trying to connect to 127.0.0.1 when config has defined 192.268.123.183

If I remove grpc_listen_address from the config (and I can see in the netstat output that 9095 is opened on all interfaces), everything works fine.

Unfortunately, I have multiple interfaces and I should only enable it on 192... network, ideally.

target: all

usage_stats:
  enabled: false

multitenancy_enabled: false

server:
  log_level: info
  http_listen_address: '192.168.123.183'
  http_listen_port: 9096
  http_tls_config:
    cert_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    key_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    client_ca_file: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    client_auth_type: RequireAndVerifyClientCert
  grpc_listen_address: '192.168.123.183'
  grpc_listen_port: 9095
  grpc_tls_config:
    cert_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    key_file: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    client_ca_file: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    client_auth_type: RequireAndVerifyClientCert

common:
  storage:
    backend: s3
    s3:
      endpoint: netapp.xxx:443
      access_key_id: xxxx
      secret_access_key: xxxxx
      insecure: false
      bucket_name: grafana-mimir

## Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  storage_prefix: blocks
  bucket_store:
    sync_interval: 5m
    ignore_blocks_within: 0s
    metadata_cache:
      bucket_index_content_ttl: 1m
      tenants_list_ttl: 1m
      tenant_blocks_list_ttl: 1m
      metafile_doesnt_exist_ttl: 1m
  tsdb:
    flush_blocks_on_shutdown: true

frontend:
  grpc_client_config:
    tls_enabled: true
    tls_server_name: mimir.ref
    tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    tls_insecure_skip_verify: false

query_scheduler:
  grpc_client_config:
    tls_enabled: true
    tls_server_name: mimir.ref
    tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    tls_insecure_skip_verify: false

ruler:
  ruler_client:
      tls_enabled: true
      tls_server_name: mimir.ref
      tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
      tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
      tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
      tls_insecure_skip_verify: false
  query_frontend:
    grpc_client_config:
      tls_enabled: true
      tls_server_name: mimir.ref
      tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
      tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
      tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
      tls_insecure_skip_verify: false

ingester_client:
  grpc_client_config:
    tls_enabled: true
    tls_server_name: mimir.ref
    tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    tls_insecure_skip_verify: false

frontend_worker:
  query_scheduler_grpc_client_config:
    tls_enabled: true
    tls_server_name: mimir.ref
    tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    tls_insecure_skip_verify: false
  grpc_client_config:
    tls_enabled: true
    tls_server_name: mimir.ref
    tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
    tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
    tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
    tls_insecure_skip_verify: false

querier:
  lookback_delta: 5m ## after this period, series is considered stale (default 5m in prometheus)

limits:
  max_global_series_per_user: 0 ## disable default per user limits
  out_of_order_time_window: 30m ## Non-zero value enables out-of-order samples
  compactor_block_upload_enabled: true
  compactor_blocks_retention_period: 1y ## Delete from storage metrics data older than configured

compactor:
  cleanup_interval: 5m

## Cluster settings
memberlist:
  randomize_node_name: false
  bind_addr: ['192.168.123.183']
  bind_port: 7946
  join_members: [mimir2,mimir3,mimir1]
  tls_enabled: true
  tls_server_name: mimir.ref
  tls_cert_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.crt
  tls_key_path: /etc/pki/ca-trust/source/anchors/xxx_mimir3.key
  tls_ca_path: /etc/pki/ca-trust/source/anchors/xxx_ca.crt
  tls_insecure_skip_verify: false
  cluster_label: mimir_group

[dimitarvdimitrov]

I'm not sure why its trying to connect to 127.0.0.1 when config has defined 192.268.123.183

In monolithic mode, all replicas internally run multiple components. In this particular case I believe it's the query-scheduler and the querier in the same monolithic process. Communication still happens over gRPC, hence the need to reach 127.0.0.1

[wally007]

I'm not sure why its trying to connect to 127.0.0.1 when config has defined 192.268.123.183

In monolithic mode, all replicas internally run multiple components. In this particular case I believe it's the query-scheduler and the querier in the same monolithic process. Communication still happens over gRPC, hence the need to reach 127.0.0.1

So essentially, in monolithic mode, there is no way to restrict GRPC to an interface ?

grpc_listen_address is not a list and if it needs 127.0.0.1 + some LAN interface - the only option is to open GRCP to all.

[dimitarvdimitrov]

So essentially, in monolithic mode, there is no way to restrict GRPC to an interface ?

I think that's the case.

grpc_listen_address is not a list and if it needs 127.0.0.1 + some LAN interface - the only option is to open GRCP to all.

I'm not that familiar with network policies (and your network provider) to be of much help here. Sorry

[wally007]

I am using on-prem vCenter, with couple of NICs attached to each mimir VM instance.

Idea was to restrict the GRPC to a single interface - which isnt possible (unlike http and memberlist address config)

Thanks for all your help. This isnt blocking issue.
Will proceed with the POC.