BackgroundService failed Resource Discovery

[jmnbc]

Report

I'm trying to run Resource Discovery, but it is failing on startup with the following error:

The configuration appears as valid and passes all the checks.

After the initialisation phase, the Resource Discovery fails with the following message:

Expected Behavior

The Resource Discovery BackgroundService to start

Actual Behavior

BackgroundService failed System.Threading.Tasks.TaskCanceledException: A task was canceled. at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37

Steps to Reproduce the Problem

1. Deploy Resource Discovery to AKS with Helm

Component

Resource Discovery

Version

0.9.1

Configuration

Configuration:

# Default values for promitor-agent-resource-discovery.
# This is a YAML-formatted file.

replicaCount: 1

image:
  repository: containers.promitor.io/tomkerkhove/promitor-agent-resource-discovery
  pullPolicy: Always
  tag:


cache:
  enabled: true
  durationInMinutes: 5

prometheus:
  enabled: true
  enableServiceDiscovery: true
  serviceMonitor:
    enabled: false
    namespace: ""
    labels: {}
    interval: 60s
    timeout: 10s

telemetry:
  defaultLogLevel: "Error"
  applicationInsights:
    enabled: false
  containerLogs:
    enabled: true
    logLevel: "Error"

## Metric Declaration YAML
azureLandscape:
  cloud: Global
  tenantId: [REDACTED]
  subscriptions:
    - [REDACTED]
resourceDiscoveryGroups:
  - name: kubernetes-service-landscape
    type: KubernetesService
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: application-gateway-landscape
    type: ApplicationGateway
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: container-registry-landscape
    type: ContainerRegistry
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: key-vault-landscape
    type: KeyVault
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
  - name: load-balancer-landscape
    type: LoadBalancer
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: network-interface-landscape
    type: NetworkInterface
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: virtual-machine-scale-set-landscape
    type: VirtualMachineScaleSet
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
  - name: virtual-network-landscape
    type: VirtualNetwork
    criteria:
      include:
        resourceGroups:
          - [REDACTED]
          - [REDACTED]
          - [REDACTED]

deployment:
  env:
    extra: []

Logs

[14:25:48 INF] Booting up Promitor v0.9.1 running .NET .NET 7.0.5 - Thank you for using Promitor!
[14:25:48 INF] Running .NET 7.0.5 on Linux (alpine.3.17-x64 | Linux 5.15.0-1042-azure #49-Ubuntu SMP Tue Jul 11 17:28:46 UTC 2023).
[14:25:48 INF] Starting validation of Promitor setup
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Name                      │ Outcome │ Details                        ┃
┠───────────────────────────┼─────────┼────────────────────────────────┨
┃ Azure Landscape           │ Success │ Everything is well-configured. ┃
┃ Resource Discovery Groups │ Success │ Everything is well-configured. ┃
┃ Azure Authentication      │ Success │ Everything is well-configured. ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
                               Validation                               
[14:25:48 INF] Promitor configuration is valid, we are good to go.
[14:25:48 INF] Here's an overview of what was configured:
╭────────────────────────────────┬─────────────┬───────────────────────────────╮
│ Tenant Id                      │ Azure Cloud │ Subscriptions                 │
├────────────────────────────────┼─────────────┼───────────────────────────────┤
│ [REDACTED]                     │ Global      │ -                             │
│                                │             │ [REDACTED]                    │
│                                │             │                               │
╰────────────────────────────────┴─────────────┴───────────────────────────────╯
                                Azure Landscape                                 
╭──────────────────────────┬─────────────────────────┬─────────────────────────╮
│ Name                     │ Resource Type           │ Is Include Criteria     │
│                          │                         │ Configured?             │
├──────────────────────────┼─────────────────────────┼─────────────────────────┤
│ kubernetes-service-lands │ Kubernetes Service      │ Yes                     │
│ cape                     │                         │                         │
│ application-gateway-land │ Application Gateway     │ Yes                     │
│ scape                    │                         │                         │
│ container-registry-lands │ Container Registry      │ Yes                     │
│ cape                     │                         │                         │
│ key-vault-landscape      │ Key Vault               │ Yes                     │
│ load-balancer-landscape  │ Load Balancer           │ Yes                     │
│ network-interface-landsc │ Network Interface       │ Yes                     │
│ ape                      │                         │                         │
│ virtual-machine-scale-se │ Virtual Machine Scale   │ Yes                     │
│ t-landscape              │ Set                     │                         │
│ virtual-network-landscap │ Virtual Network         │ Yes                     │
│ e                        │                         │                         │
╰──────────────────────────┴─────────────────────────┴─────────────────────────╯
                           Resource Discovery Groups                            
[14:25:49 INF] [Job][Azure Subscription Discovery] is running under time zone: UTC
[14:25:49 INF] [Job][Azure Resource Group Discovery] is running under time zone: UTC
[14:25:49 INF] Adding Prometheus sink to expose on /metrics
[14:25:50 ERR] BackgroundService failed
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
   at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)
[14:25:50 FTL] Promitor Discovery Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.Net.Sockets.SocketException (13): Permission denied
   at System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress socketAddress)
   at System.Net.Sockets.Socket.Bind(EndPoint localEP)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportOptions.CreateDefaultBoundListenSocket(EndPoint endpoint)
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketConnectionListener.Bind()
   at Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.SocketTransportFactory.BindAsync(EndPoint endpoint, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Infrastructure.TransportManager.BindAsync(EndPoint endPoint, ConnectionDelegate connectionDelegate, EndpointConfig endpointConfig, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.<>c__DisplayClass30_0`1.<<StartAsync>g__OnBind|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindEndpointAsync(ListenOptions endpoint, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.ListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.AnyIPListenOptions.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.AddressesStrategy.BindAsync(AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.AddressBinder.BindAsync(IEnumerable`1 listenOptions, AddressBindContext context, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Promitor.Agents.ResourceDiscovery.Program.Main(String[] args) in /src/Promitor.Agents.ResourceDiscovery/Program.cs:line 50
[14:25:50 FTL] The HostOptions.BackgroundServiceExceptionBehavior is configured to StopHost. A BackgroundService has thrown an unhandled exception, and the IHost instance is stopping. To avoid this behavior, configure this to Ignore; however the BackgroundService will not be restarted.
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at CronScheduler.Extensions.Internal.SchedulerHostedService.ExecuteAsync(CancellationToken stoppingToken) in C:\projects\cronscheduler-aspnetcore\src\CronScheduler.Extensions\Internal\SchedulerHostedService.cs:line 37
   at Microsoft.Extensions.Hosting.Internal.Host.TryExecuteBackgroundServiceAsync(BackgroundService backgroundService)

Platform

Microsoft Azure

Contact Details

[email protected]

[MattiasAng]

@jmnbc This is due to changes to distroless and what securityContext settings are required.

Upgrade to the latest Helm chart version and it should work.