Nomad Autoscaler Idle forever with context.canceled error

[bernardoVale]

I couldn't find a way to reproduce this error, but sometimes, autoscaler gets stuck and returns context canceled to all policy evaluations:

2023-09-19T18:08:24.247Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=af2adcdd-ffc6-8c54-e71d-dbbf55543a77 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.349Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=9418971a-8261-cbd7-7730-5f8748f455d0 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.447Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=04db11b8-6242-c03d-cb93-010c35e35285 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.551Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=3e2681aa-68d2-41ae-f4e0-cd1dcd0b8ba3 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.848Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=3c10001b-0402-743b-5bfd-73577527d9ac error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:24.948Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=236fcef8-6a4f-be5d-ae61-fd56eabfe92f error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.148Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=fa573b85-9ab6-952c-8deb-611a59f64d32 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.249Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8bf225c1-5a57-7ce1-d13e-ce63f7de45de error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.345Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8ea0ab80-bbaa-35b0-7a50-69bc38a719fb error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:25.348Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=ee9ea733-6563-d=c3767a43-77c9-d8a4-3101-1bd6a6cb6c8f error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:26.248Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=8ff31dca-fce8-0624-3669-2839a768752e error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:26.348Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=df860074-418c-fb31-d2c4-e970b1f95d39 error="rpc error: code = Canceled desc = context canceled"
2023-09-19T18:08:52.520Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=32725454-2121-9bea-29b5-f81405eeb3e2 error="rpc error: code = Canceled desc = context canceled"

Each eval loop, it prints one context canceled error per policy. Once it gets into this state, only a restart fixes it.

Here's the SIGABRT dump:

SIGABRT: abort

PC=0x46d261 m=0 sigcode=0

goroutine 0 [idle]:

runtime.futex()

	runtime/sys_linux_amd64.s:519 +0x21

runtime.futexsleep(0xc000052000, 0x57000, 0x1970b5f)

	runtime/os_linux.go:44 +0x36

runtime.notesleep(0x2b6ef30)

	runtime/lock_futex.go:160 +0x87

runtime.mPark()

	runtime/proc.go:1441 +0x2a

runtime.stopm()

	runtime/proc.go:2408 +0x78

runtime.findrunnable()

	runtime/proc.go:2984 +0x865

runtime.schedule()

	runtime/proc.go:3367 +0x239

runtime.goexit0(0xc00099c9c0)

	runtime/proc.go:3691 +0x21e

runtime.mcall()

	runtime/asm_amd64.s:307 +0x43



goroutine 1 [chan receive, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).handleSignals(0xc0004299d0)

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:270 +0xe5

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).Run(0xc0004299d0)

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:100 +0x446

github.com/hashicorp/nomad-autoscaler/command.(*AgentCommand).Run(0xc0002a7860, {0xc0001a8020, 0x203000, 0x0})

	github.com/hashicorp/nomad-autoscaler/command/agent.go:333 +0x8e5

github.com/mitchellh/cli.(*CLI).Run(0xc0001e9b80)

	github.com/mitchellh/[email protected]/cli.go:262 +0x5f8

main.main()

	github.com/hashicorp/nomad-autoscaler/main.go:26 +0x24d



goroutine 5 [select]:

go.opencensus.io/stats/view.(*worker).start(0xc000380380)

	[email protected]/stats/view/worker.go:276 +0xb9

created by go.opencensus.io/stats/view.init.0

	[email protected]/stats/view/worker.go:34 +0x92



goroutine 7 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e33d8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00020cd00, 0x4172e6, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Accept(0xc00020cd00)

	internal/poll/fd_unix.go:402 +0x22c

net.(*netFD).accept(0xc00020cd00)

	net/fd_unix.go:173 +0x35

net.(*TCPListener).accept(0xc00000c9d8)

	net/tcpsock_posix.go:140 +0x28

net.(*TCPListener).Accept(0xc00000c9d8)

	net/tcpsock.go:262 +0x3d

net/http.(*Server).Serve(0xc0003ece00, {0x1e39d40, 0xc00000c9d8})

	net/http/server.go:3002 +0x394

github.com/hashicorp/nomad-autoscaler/agent/http.(*Server).Start(0xc0001b1b80)

	github.com/hashicorp/nomad-autoscaler/agent/http/server.go:124 +0xd2

created by github.com/hashicorp/nomad-autoscaler/command.(*AgentCommand).Run

	github.com/hashicorp/nomad-autoscaler/command/agent.go:330 +0x896



goroutine 8 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e3108, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0005bf7a0, 0xc000640000, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0005bf7a0, {0xc000640000, 0x10000, 0x10000})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc0001a2ae0, {0xc000640000, 0x10000c00063bde8, 0x7f4c8d23cd40})

	os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc00063bf40)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc00063bf40, 0x0)

	bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc00063bf40)

	bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc0003e2960, {0x1e17260, 0xc0001a2ae0})

	github.com/hashicorp/[email protected]/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:601 +0x14af



goroutine 9 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0x0)

	runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x0)

	sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

	github.com/hashicorp/[email protected]/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:604 +0x152f



goroutine 10 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e32e8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0005bf6e0, 0xc000256025, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0005bf6e0, {0xc000256025, 0xfdb, 0xfdb})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc0001a2ad0, {0xc000256025, 0xc00060eea8, 0x40592d})

	os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc00060ef40)

	bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

	github.com/hashicorp/[email protected]/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:645 +0x15f4



goroutine 97 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

	github.com/hashicorp/[email protected]/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

	github.com/hashicorp/[email protected]/client.go:661 +0x85



goroutine 98 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0006d6080)

	google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

	google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22f



goroutine 32 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

	google.golang.org/[email protected]/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

	google.golang.org/[email protected]/stream.go:340 +0xb65



goroutine 100 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

	github.com/hashicorp/[email protected]/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0006d0140)

	github.com/hashicorp/[email protected]/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:62 +0x287



goroutine 101 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000316240)

	google.golang.org/[email protected]/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

	google.golang.org/[email protected]/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc000318000, {0x1aab320, 0xc000310140}, 0x8)

	google.golang.org/[email protected]/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

	google.golang.org/[email protected]/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc000316000, 0xc000713e90, 0xc000713e60)

	google.golang.org/[email protected]/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc000316000, {0x1aab320, 0xc000310140})

	google.golang.org/[email protected]/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc00018b940)

	github.com/hashicorp/[email protected]/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc0006d6200)

	github.com/hashicorp/[email protected]/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:63 +0x2c9



goroutine 11 [IO wait, 23 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e3018, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00020cf00, 0xc0005e0000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc00020cf00, {0xc0005e0000, 0x8000, 0x8000})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc00020cf00, {0xc0005e0000, 0x60100000000, 0x8})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0001a2af0, {0xc0005e0000, 0x4693ae, 0x439c47})

	net/net.go:183 +0x45

bufio.(*Reader).Read(0xc0005bfa40, {0xc0003ed000, 0x9, 0xc00062dd18})

	bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc0005bfa40}, {0xc0003ed000, 0x9, 0x9}, 0x9)

	io/io.go:328 +0x9a

io.ReadFull(...)

	io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc0003ed000, 0x9, 0xc00062dd6f}, {0x1e13d00, 0xc0005bfa40})

	golang.org/x/[email protected]/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003ecfc0)

	golang.org/x/[email protected]/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00000a1e0)

	google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x194f



goroutine 12 [select, 23 minutes]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0005de0a0, 0x1)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0005bfbc0)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

	google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x1fa5



goroutine 113 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

	github.com/hashicorp/[email protected]/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

	github.com/hashicorp/[email protected]/grpc_broker.go:179 +0x17a



goroutine 129 [IO wait, 2414 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2e38, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0007321e0, 0xc000010000, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0007321e0, {0xc000010000, 0x10000, 0x10000})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc00071c050, {0xc000010000, 0xc00299ade8, 0x4bc716})

	os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc00299af40)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc00299af40, 0xb0)

	bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc00299af40)

	bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc00075e000, {0x1e17260, 0xc00071c050})

	github.com/hashicorp/[email protected]/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:601 +0x14af



goroutine 130 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0xc000619f20)

	runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x1)

	sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

	github.com/hashicorp/[email protected]/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:604 +0x152f



goroutine 131 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e31f8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000732120, 0xc000252025, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000732120, {0xc000252025, 0xfdb, 0xfdb})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc00071c040, {0xc000252025, 0xc0007796a8, 0x40592d})

	os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc000779740)

	bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

	github.com/hashicorp/[email protected]/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:645 +0x15f4



goroutine 82 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

	github.com/hashicorp/[email protected]/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

	github.com/hashicorp/[email protected]/client.go:661 +0x85



goroutine 83 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00063e140)

	google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

	google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22f



goroutine 86 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

	google.golang.org/[email protected]/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

	google.golang.org/[email protected]/stream.go:340 +0xb65



goroutine 14 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

	github.com/hashicorp/[email protected]/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0005de230)

	github.com/hashicorp/[email protected]/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:62 +0x287



goroutine 15 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00002e120)

	google.golang.org/[email protected]/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

	google.golang.org/[email protected]/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc0003858c0, {0x1aab320, 0xc0000f24b0}, 0x8)

	google.golang.org/[email protected]/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

	google.golang.org/[email protected]/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc0005d06c0, 0xc00070fe90, 0xc00070fe60)

	google.golang.org/[email protected]/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc0005d06c0, {0x1aab320, 0xc0000f24b0})

	google.golang.org/[email protected]/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc000634800)

	github.com/hashicorp/[email protected]/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc0005f8180)

	github.com/hashicorp/[email protected]/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:63 +0x2c9



goroutine 84 [IO wait, 13 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2d48, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000380200, 0xc000144000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000380200, {0xc000144000, 0x8000, 0x8000})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000380200, {0xc000144000, 0x60100000000, 0x8})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000632068, {0xc000144000, 0x4693ae, 0x439c47})

	net/net.go:183 +0x45

bufio.(*Reader).Read(0xc000650300, {0xc00002a040, 0x9, 0xc0005d6d18})

	bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc000650300}, {0xc00002a040, 0x9, 0x9}, 0x9)

	io/io.go:328 +0x9a

io.ReadFull(...)

	io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc00002a040, 0x9, 0xc0005d6d6f}, {0x1e13d00, 0xc000650300})

	golang.org/x/[email protected]/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc00002a000)

	golang.org/x/[email protected]/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00002c000)

	google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x194f



goroutine 85 [select, 13 minutes]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000f2370, 0x1)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000650420)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

	google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x1fa5



goroutine 87 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

	github.com/hashicorp/[email protected]/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

	github.com/hashicorp/[email protected]/grpc_broker.go:179 +0x17a



goroutine 162 [select, 2338 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0005f8500)

	google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

	google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22f



goroutine 170 [chan receive]:

github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin.(*TargetPlugin).garbageCollectionLoop(0xc0005fccc0)

	github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin/plugin.go:213 +0xc9

created by github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin.(*TargetPlugin).SetConfig

	github.com/hashicorp/nomad-autoscaler/plugins/builtin/target/nomad/plugin/plugin.go:94 +0xef



goroutine 88 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e2898, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000312540, 0xc00033e000, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000312540, {0xc00033e000, 0x10000, 0x10000})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc00000e048, {0xc00033e000, 0x10000c0009cbde8, 0x7f4c8d07d5c0})

	os/file.go:119 +0x5e

bufio.(*Reader).fill(0xc0009cbf40)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).ReadSlice(0xc0009cbf40, 0x70)

	bufio/bufio.go:360 +0x2f

bufio.(*Reader).ReadLine(0xc0009cbf40)

	bufio/bufio.go:389 +0x27

github.com/hashicorp/go-plugin.(*Client).logStderr(0xc00042a3c0, {0x1e17260, 0xc00000e048})

	github.com/hashicorp/[email protected]/client.go:956 +0x2aa

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:601 +0x14af



goroutine 89 [semacquire, 28822 minutes]:

sync.runtime_Semacquire(0x0)

	runtime/sema.go:56 +0x25

sync.(*WaitGroup).Wait(0x0)

	sync/waitgroup.go:130 +0x71

github.com/hashicorp/go-plugin.(*Client).Start.func2()

	github.com/hashicorp/[email protected]/client.go:617 +0xc8

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:604 +0x152f



goroutine 90 [IO wait, 28822 minutes]:

internal/poll.runtime_pollWait(0x7f4c8d2e2c58, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000312480, 0xc000026025, 0x1)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000312480, {0xc000026025, 0xfdb, 0xfdb})

	internal/poll/fd_unix.go:167 +0x25a

os.(*File).read(...)

	os/file_posix.go:32

os.(*File).Read(0xc00000e038, {0xc000026025, 0xc000718ea8, 0x40592d})

	os/file.go:119 +0x5e

bufio.(*Scanner).Scan(0xc000718f40)

	bufio/scan.go:215 +0x865

github.com/hashicorp/go-plugin.(*Client).Start.func3()

	github.com/hashicorp/[email protected]/client.go:650 +0x12f

created by github.com/hashicorp/go-plugin.(*Client).Start

	github.com/hashicorp/[email protected]/client.go:645 +0x15f4



goroutine 114 [chan receive, 28822 minutes]:

github.com/hashicorp/go-plugin.(*Client).Start.func4.1()

	github.com/hashicorp/[email protected]/client.go:663 +0x7f

created by github.com/hashicorp/go-plugin.(*Client).Start.func4

	github.com/hashicorp/[email protected]/client.go:661 +0x85



goroutine 115 [select, 28822 minutes]:

google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00063e580)

	google.golang.org/[email protected]/balancer_conn_wrappers.go:112 +0x79

created by google.golang.org/grpc.newCCBalancerWrapper

	google.golang.org/[email protected]/balancer_conn_wrappers.go:73 +0x22f



goroutine 149 [syscall, 28822 minutes]:

os/signal.signal_recv()

	runtime/sigqueue.go:169 +0x98

os/signal.loop()

	os/signal/signal_unix.go:24 +0x19

created by os/signal.Notify.func1.1

	os/signal/signal.go:151 +0x2c



goroutine 117 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).Recv(0x0)

	github.com/hashicorp/[email protected]/grpc_broker.go:231 +0x6d

github.com/hashicorp/go-plugin.(*GRPCBroker).Run(0xc0000f2870)

	github.com/hashicorp/[email protected]/grpc_broker.go:411 +0x48

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:62 +0x287



goroutine 118 [select, 28822 minutes]:

google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc00002e7e0)

	google.golang.org/[email protected]/internal/transport/transport.go:324 +0x85

google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)

	google.golang.org/[email protected]/internal/transport/transport.go:339

google.golang.org/grpc.(*csAttempt).recvMsg(0xc000384a50, {0x1aab320, 0xc0003104b0}, 0x8)

	google.golang.org/[email protected]/stream.go:969 +0xbb

google.golang.org/grpc.(*clientStream).RecvMsg.func1(0x203000)

	google.golang.org/[email protected]/stream.go:834 +0x25

google.golang.org/grpc.(*clientStream).withRetry(0xc0006d2900, 0xc0001bfe90, 0xc0001bfe60)

	google.golang.org/[email protected]/stream.go:688 +0x2f6

google.golang.org/grpc.(*clientStream).RecvMsg(0xc0006d2900, {0x1aab320, 0xc0003104b0})

	google.golang.org/[email protected]/stream.go:833 +0x11f

github.com/hashicorp/go-plugin/internal/plugin.(*gRPCBrokerStartStreamClient).Recv(0xc0000eca40)

	github.com/hashicorp/[email protected]/internal/plugin/grpc_broker.pb.go:149 +0x4c

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream(0xc00063e6c0)

	github.com/hashicorp/[email protected]/grpc_broker.go:194 +0x1dd

created by github.com/hashicorp/go-plugin.newGRPCClient

	github.com/hashicorp/[email protected]/grpc_client.go:63 +0x2c9



goroutine 91 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e27a8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0006d8400, 0xc0002d6000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0006d8400, {0xc0002d6000, 0x8000, 0x8000})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc0006d8400, {0xc0002d6000, 0x1060100000000, 0x8})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00000e0f0, {0xc0002d6000, 0x97b8b0, 0x1})

	net/net.go:183 +0x45

bufio.(*Reader).Read(0xc000312720, {0xc00075c200, 0x9, 0x18})

	bufio/bufio.go:227 +0x1b4

io.ReadAtLeast({0x1e13d00, 0xc000312720}, {0xc00075c200, 0x9, 0x9}, 0x9)

	io/io.go:328 +0x9a

io.ReadFull(...)

	io/io.go:347

golang.org/x/net/http2.readFrameHeader({0xc00075c200, 0x9, 0x624d25f18c414}, {0x1e13d00, 0xc000312720})

	golang.org/x/[email protected]/http2/frame.go:237 +0x6e

golang.org/x/net/http2.(*Framer).ReadFrame(0xc00075c1c0)

	golang.org/x/[email protected]/http2/frame.go:498 +0x95

google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00002c1e0)

	google.golang.org/[email protected]/internal/transport/http2_client.go:1498 +0x41f

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:365 +0x194f



goroutine 92 [select]:

google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000310370, 0x1)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:407 +0x11b

google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0003128a0)

	google.golang.org/[email protected]/internal/transport/controlbuf.go:534 +0x85

google.golang.org/grpc/internal/transport.newHTTP2Client.func3()

	google.golang.org/[email protected]/internal/transport/http2_client.go:415 +0x65

created by google.golang.org/grpc/internal/transport.newHTTP2Client

	google.golang.org/[email protected]/internal/transport/http2_client.go:413 +0x1fa5



goroutine 93 [select, 28822 minutes]:

google.golang.org/grpc.newClientStreamWithParams.func4()

	google.golang.org/[email protected]/stream.go:341 +0x98

created by google.golang.org/grpc.newClientStreamWithParams

	google.golang.org/[email protected]/stream.go:340 +0xb65



goroutine 94 [select, 28822 minutes]:

github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream.func1()

	github.com/hashicorp/[email protected]/grpc_broker.go:181 +0x10e

created by github.com/hashicorp/go-plugin.(*gRPCBrokerClientImpl).StartStream

	github.com/hashicorp/[email protected]/grpc_broker.go:179 +0x17a



goroutine 150 [select, 28822 minutes]:

github.com/armon/go-metrics.(*InmemSignal).run(0xc0005f8c40)

	github.com/armon/[email protected]/inmem_signal.go:64 +0x6c

created by github.com/armon/go-metrics.NewInmemSignal

	github.com/armon/[email protected]/inmem_signal.go:38 +0x174



goroutine 152 [sleep]:

time.Sleep(0x3b9aca00)

	runtime/time.go:193 +0x12e

github.com/armon/go-metrics.(*Metrics).collectStats(0xc0001fa2d0)

	github.com/armon/[email protected]/metrics.go:234 +0x25

created by github.com/armon/go-metrics.New

	github.com/armon/[email protected]/start.go:84 +0x17b



goroutine 154 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e640, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 155 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e690, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 156 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e6e0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 157 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e730, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 158 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e780, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 159 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e7d0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 160 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e820, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 177 [select]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e870, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 178 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e8c0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 179 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be7579, 0xa})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e910, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:128 +0x25f



goroutine 180 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e960, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 181 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089e9b0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 182 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ea00, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 183 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ea50, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 184 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eaa0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 185 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eaf0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 186 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eb40, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 187 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089eb90, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 188 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ebe0, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 189 [select, 28822 minutes]:

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).waitForWork(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:246 +0x1e5

github.com/hashicorp/nomad-autoscaler/policyeval.(*Broker).Dequeue(0xc0008b4780, {0x1e4cfb0, 0xc0001a7cc0}, {0x1be2f4a, 0x7})

	github.com/hashicorp/nomad-autoscaler/policyeval/broker.go:176 +0x19a

github.com/hashicorp/nomad-autoscaler/policyeval.(*BaseWorker).Run(0xc00089ec30, {0x1e4cfb0, 0xc0001a7cc0})

	github.com/hashicorp/nomad-autoscaler/policyeval/base_worker.go:60 +0xb6

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).initWorkers

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:134 +0x35b



goroutine 190 [select]:

github.com/hashicorp/nomad-autoscaler/agent.(*Agent).runEvalHandler(0xc0004299d0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008b4720)

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:106 +0xb2

created by github.com/hashicorp/nomad-autoscaler/agent.(*Agent).Run

	github.com/hashicorp/nomad-autoscaler/agent/agent.go:97 +0x439



goroutine 4504214 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc002c98240)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4543502 [select]:

net/http.(*persistConn).roundTrip(0xc001970000, 0xc0022e0340)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007cf400)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007cf400, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x417901, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007cf400, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007cf400)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000b91e40)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x24, {0xc000575b80, 0xc000b6e4c0}, {0x1879000, 0xc0001b9340}, 0xc13a9754473257de)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00102e7b0, {0xc0022d1200, 0x24}, 0xc001fc0480)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4504291 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2f2d8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000380180, 0xc002416600, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000380180, {0xc002416600, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000380180, {0xc002416600, 0xc002416605, 0x28c})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000a20008, {0xc002416600, 0x6, 0xc0019c87f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc000630048, {0xc002416600, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000192278, {0x1e13f00, 0xc000630048})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000192000, {0x1e16e20, 0xc000a20008}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000192000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000192000, {0xc002458000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc00072e000, {0xc002458000, 0xc002642960, 0xc0019c8d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002e830e0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002e830e0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc00072e000)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4504310 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0006d26c0)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4543255 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc0020da000, 0xc0015281c0)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0009aa500)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0009aa500, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0009aa500, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0009aa500)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001644d10)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0xc001986748, {0xc00110ed40, 0x2ba0fe0}, {0x1879000, 0xc000bba770}, 0xc001838058)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0019867b0, {0xc0022d0d50, 0x24}, 0xc001fc39a0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4414570 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf06cf0, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000296180, 0xc002310000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000296180, {0xc002310000, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000296180, {0xc002310000, 0xc002310005, 0x267})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001724000, {0xc002310000, 0x6, 0xc001f4a7f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0020ee0d8, {0xc002310000, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc001730278, {0x1e13f00, 0xc0020ee0d8})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001730000, {0x1e16e20, 0xc001724000}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001730000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001730000, {0xc002338000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0020da000, {0xc002338000, 0xc000314c00, 0xc001f4ad30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002b8cf00)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002b8cf00, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0020da000)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 1552064 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc001789580}, {{0xc0022d0c60, 0x24}, 0xc000b311a0, 0xc000b312c0, 0xc000b31140})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1553913 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80c80}, {{0xc0022d0e10, 0x24}, 0xc002546780, 0xc0025468a0, 0xc002546720})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1554082 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorIDs(0xc0008b0400, {0x1e4cfb0, 0xc000c31440}, {0xc002075140, 0xc0020750e0})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:113 +0x26d

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:67 +0x1f3



goroutine 4542337 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc000ca2120)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 1554093 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450780, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008c6b00)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0e10, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 4543253 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc001970120, 0xc001332180)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc00118dc00)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc00118dc00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc00118dc00, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc00118dc00)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0001616b0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4d80, 0x62493e4ec443e}, {0x1879000, 0xc00067c1c0}, 0x62493e4ec443e)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0011457b0, {0xc0022d0e40, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1554162 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0022c2300}, {{0xc0022d0f30, 0x24}, 0xc0025472c0, 0xc0025473e0, 0xc002547260})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1554091 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450640, {0x1e4cfb0, 0xc0001a7cc0}, 0xc001980840)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0db0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 1553755 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc001111040}, {{0xc0022d0de0, 0x24}, 0xc0025465a0, 0xc0025466c0, 0xc002546540})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1554055 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc002503680}, {{0xc0022d0d50, 0x24}, 0xc000b31b60, 0xc000b31c80, 0xc000b31b00})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1553845 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc00063fe80}, {{0xc0022d0f90, 0x24}, 0xc002547680, 0xc0025477a0, 0xc002547620})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1553923 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0004abe40}, {{0xc0022d10b0, 0x24}, 0xc001793a40, 0xc001793b60, 0xc0017939e0})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4504279 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2fe18, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000528080, 0xc002304c00, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000528080, {0xc002304c00, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000528080, {0xc002304c00, 0xc002304c05, 0x279})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001100008, {0xc002304c00, 0x6, 0xc0026107f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0018ca078, {0xc002304c00, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000130cf8, {0x1e13f00, 0xc0018ca078})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000130a80, {0x1e16e20, 0xc001100008}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000130a80, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000130a80, {0xc0024d6000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0006d2240, {0xc0024d6000, 0xc001f94d20, 0xc002610d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0005c5da0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0005c5da0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0006d2240)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 1554092 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024506e0, {0x1e4cfb0, 0xc0001a7cc0}, 0x1e4cfb0)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0de0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 4504216 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc001970120)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 1554084 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024501e0, {0x1e4cfb0, 0xc0001a7cc0}, 0x1e4cfb0)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0c60, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 4214783 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8ce2f788, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000f18300, 0xc002300000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000f18300, {0xc002300000, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000f18300, {0xc002300000, 0xc002300005, 0x26b})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0008c8020, {0xc002300000, 0x6, 0xc001e267f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc00000c300, {0xc002300000, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0018b6278, {0x1e13f00, 0xc00000c300})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0018b6000, {0x1e16e20, 0xc0008c8020}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0018b6000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0018b6000, {0xc002494000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002b20000, {0xc002494000, 0xc00112c240, 0xc001e26d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0004ce6c0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0004ce6c0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002b20000)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4542326 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8ce2fa58, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000e12400, 0xc002483900, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000e12400, {0xc002483900, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000e12400, {0xc002483900, 0xc002483905, 0x26c})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00091a0a0, {0xc002483900, 0x6, 0xc0029967f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0013b40a8, {0xc002483900, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0016365f8, {0x1e13f00, 0xc0013b40a8})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001636380, {0x1e16e20, 0xc00091a0a0}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001636380, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001636380, {0xc000af6000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc00093e000, {0xc000af6000, 0xc000181d40, 0xc002996d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0002b9a40)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0002b9a40, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc00093e000)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4543171 [select, 5 minutes]:

net/http.(*persistConn).roundTrip(0xc0006ec360, 0xc000b6e140)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000170400)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000170400, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000170400, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000170400)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0014ac840)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc000d92880, 0x62470f845a44d}, {0x1879000, 0xc00067c0e0}, 0x62470f845a44d)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001a71fb0, {0xc0022d10e0, 0x24}, 0xc0006d2120)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1554098 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450b40, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000319290)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0f00, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 1554130 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000915a40}, {{0xc0022d0e70, 0x24}, 0xc002546b40, 0xc002546c60, 0xc002546ae0})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4542313 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0005f8c00}, {{0xc0013fd200, 0x24}, 0xc0010da480, 0xc0010da600, 0xc0010da420})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 1554097 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450aa0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0015246e0)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0ed0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 1554087 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc0024503c0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc001860210)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0cf0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 1554112 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002451400, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000161290)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d11a0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 4543338 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002c98120, 0xc00063e440)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c5300)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c5300, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c5300, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c5300)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000918bb0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x0, {0xc0015850c0, 0x14c8942}, {0x1879000, 0xc000589570}, 0x624afd586b04d)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0033b9fb0, {0xc0022d1050, 0x24}, 0xc002e8a220)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4543219 [select, 5 minutes]:

net/http.(*persistConn).roundTrip(0xc002b20120, 0xc002502300)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000302b00)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000302b00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000302b00, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000302b00)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc00113a9a0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00188cfc0, 0x62493ba612df8}, {0x1879000, 0xc00049a4d0}, 0x62493ba612df8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0018187b0, {0xc0022d0db0, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1553869 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000735b40}, {{0xc0022d0e40, 0x24}, 0xc002546960, 0xc002546a80, 0xc002546900})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4272239 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8d2e2a78, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000544300, 0xc002414000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000544300, {0xc002414000, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000544300, {0xc002414000, 0xc002414005, 0x266})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000112060, {0xc002414000, 0x6, 0xc0019c47f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc000132078, {0xc002414000, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000532278, {0x1e13f00, 0xc000132078})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000532000, {0x1e16e20, 0xc000112060}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000532000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000532000, {0xc00243a000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0005d05a0, {0xc00243a000, 0xc00112c2a0, 0xc0019c4d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc000650ea0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc000650ea0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0005d05a0)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4542385 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf06b10, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc000f18280, 0xc002314c00, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc000f18280, {0xc002314c00, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc000f18280, {0xc002314c00, 0xc002314c05, 0x58a})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00071c060, {0xc002314c00, 0xc000217500, 0xc0003b47f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0008ae090, {0xc002314c00, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0014813f8, {0x1e13f00, 0xc0008ae090})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001481180, {0x1e16e20, 0xc00071c060}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001481180, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001481180, {0xc000af9000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002a00240, {0xc000af9000, 0xc000426de0, 0xc0003b4d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002e83380)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002e83380, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002a00240)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4487210 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc00216e000)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4543367 [select]:

net/http.(*persistConn).roundTrip(0xc0005d05a0, 0xc000b80140)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000c98200)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000c98200, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000c98200, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000c98200)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc00113ab00)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0xc00176f748, {0xc00110e3c0, 0x14c8942}, {0x1879000, 0xc0003fe0e0}, 0xc000536118)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00176f7b0, {0xc0022d0f00, 0x24}, 0xc002e8a220)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1553949 [select]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0017d7700}, {{0xc0022d0f00, 0x24}, 0xc0025470e0, 0xc002547200, 0xc002547080})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4543307 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc00093e000, 0xc000c30300)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000bdf700)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000bdf700, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000bdf700, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000bdf700)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0006ee2c0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00147cac0, 0x62485ec8ab0d8}, {0x1879000, 0xc0002b4070}, 0x62485ec8ab0d8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc000604fb0, {0xc0013fced0, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1554106 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002451040, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000d99000)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d1080, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 1553840 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0022c2240}, {{0xc0022d0d80, 0x24}, 0xc000b31d40, 0xc000b31e60, 0xc000b31ce0})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4504293 [IO wait]:

internal/poll.runtime_pollWait(0x7f4c8cf05a30, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc0004d0100, 0xc002417900, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc0004d0100, {0xc002417900, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc0004d0100, {0xc002417900, 0xc002417905, 0x269})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc00051c010, {0xc002417900, 0x6, 0xc0024617f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc002736048, {0xc002417900, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0003b1078, {0x1e13f00, 0xc002736048})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0003b0e00, {0x1e16e20, 0xc00051c010}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0003b0e00, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0003b0e00, {0xc002474000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc001970000, {0xc002474000, 0xc002642ba0, 0xc002461d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0027c3e60)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0027c3e60, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc001970000)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4543353 [select]:

net/http.(*persistConn).roundTrip(0xc002a00120, 0xc001332140)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0009aa000)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0009aa000, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0009aa000, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0009aa000)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0001b5ef0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc001512640, 0x620faa667a5f8}, {0x1879000, 0xc0004449a0}, 0x620faa667a5f8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001022fb0, {0xc0022d1110, 0x24}, 0xc0025b2000)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4543292 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002a00000, 0xc000914180)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc002052200)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc002052200, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc002052200, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc002052200)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001fc6370)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc00188c6c0, 0x62493ba612df8}, {0x1879000, 0xc0005928c0}, 0x62493ba612df8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc0018187b0, {0xc0022d10b0, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4272240 [select]:

net/http.(*persistConn).writeLoop(0xc0005d05a0)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4504030 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8ce2f5a8, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc002e92180, 0xc00082a000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc002e92180, {0xc00082a000, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc002e92180, {0xc00082a000, 0xc00082a005, 0x27d})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc0009ac0b8, {0xc00082a000, 0x6, 0xc0020217f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0018ca060, {0xc00082a000, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc0013a6278, {0x1e13f00, 0xc0018ca060})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc0013a6000, {0x1e16e20, 0xc0009ac0b8}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc0013a6000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc0013a6000, {0xc001009000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0006d2120, {0xc001009000, 0xc000181200, 0xc002021d30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0020c4720)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0020c4720, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0006d2120)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 4504313 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf05760, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc002228780, 0xc002279300, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc002228780, {0xc002279300, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc002228780, {0xc002279300, 0xc002279305, 0x272})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc001082030, {0xc002279300, 0x6, 0xc00260c7f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0014020a8, {0xc002279300, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc000c22278, {0x1e13f00, 0xc0014020a8})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc000c22000, {0x1e16e20, 0xc001082030}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc000c22000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc000c22000, {0xc002624000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc002c98360, {0xc002624000, 0xc0010e4660, 0xc00260cd30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc002155da0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc002155da0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc002c98360)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 1553915 [select, 3 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80d80}, {{0xc0022d0f60, 0x24}, 0xc0025474a0, 0xc0025475c0, 0xc002547440})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4504312 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0006d2000)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 1553968 [select, 19 minutes]:

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run(0xc00089e5a0, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008b4720)

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:72 +0x475

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:154 +0x847



goroutine 4487206 [select, 3 minutes]:

net/http.(*persistConn).writeLoop(0xc0001ce480)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4543304 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc0006d2120, 0xc0010de180)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000ac1a00)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000ac1a00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000ac1a00, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000ac1a00)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc0003193f0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4bc0, 0x62485ec8ab0d8}, {0x1879000, 0xc000428690}, 0x62485ec8ab0d8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc000604fb0, {0xc0022d1080, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1554086 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450320, {0x1e4cfb0, 0xc0001a7cc0}, 0xc0008c6b00)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0cc0, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 3878095 [IO wait, 3 minutes]:

internal/poll.runtime_pollWait(0x7f4c8cf061b0, 0x72)

	runtime/netpoll.go:303 +0x85

internal/poll.(*pollDesc).wait(0xc00075a300, 0xc002284000, 0x0)

	internal/poll/fd_poll_runtime.go:84 +0x32

internal/poll.(*pollDesc).waitRead(...)

	internal/poll/fd_poll_runtime.go:89

internal/poll.(*FD).Read(0xc00075a300, {0xc002284000, 0x109c, 0x109c})

	internal/poll/fd_unix.go:167 +0x25a

net.(*netFD).Read(0xc00075a300, {0xc002284000, 0xc002284005, 0x76d})

	net/fd_posix.go:56 +0x29

net.(*conn).Read(0xc000e18078, {0xc002284000, 0x6, 0xc001f0f7f8})

	net/net.go:183 +0x45

crypto/tls.(*atLeastReader).Read(0xc0002e6060, {0xc002284000, 0x0, 0x409f2d})

	crypto/tls/conn.go:777 +0x3d

bytes.(*Buffer).ReadFrom(0xc001256278, {0x1e13f00, 0xc0002e6060})

	bytes/buffer.go:204 +0x98

crypto/tls.(*Conn).readFromUntil(0xc001256000, {0x1e16e20, 0xc000e18078}, 0x109c)

	crypto/tls/conn.go:799 +0xe5

crypto/tls.(*Conn).readRecordOrCCS(0xc001256000, 0x0)

	crypto/tls/conn.go:606 +0x112

crypto/tls.(*Conn).readRecord(...)

	crypto/tls/conn.go:574

crypto/tls.(*Conn).Read(0xc001256000, {0xc0022a6000, 0x1000, 0x0})

	crypto/tls/conn.go:1277 +0x16f

net/http.(*persistConn).Read(0xc0001ce360, {0xc0022a6000, 0xc001522240, 0xc001f0fd30})

	net/http/transport.go:1926 +0x4e

bufio.(*Reader).fill(0xc0023963c0)

	bufio/bufio.go:101 +0x103

bufio.(*Reader).Peek(0xc0023963c0, 0x1)

	bufio/bufio.go:139 +0x5d

net/http.(*persistConn).readLoop(0xc0001ce360)

	net/http/transport.go:2087 +0x1ac

created by net/http.(*Transport).dialConn

	net/http/transport.go:1747 +0x1e05



goroutine 1554101 [select]:

github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run(0xc002450d20, {0x1e4cfb0, 0xc0001a7cc0}, 0xc000918790)

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:118 +0x411

github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run.func1({0xc0022d0f90, 0x24})

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:116 +0x46

created by github.com/hashicorp/nomad-autoscaler/policy.(*Manager).Run

	github.com/hashicorp/nomad-autoscaler/policy/manager.go:115 +0xed3



goroutine 4214784 [select]:

net/http.(*persistConn).writeLoop(0xc002b20000)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 4543412 [select]:

net/http.(*persistConn).roundTrip(0xc002c98000, 0xc001110240)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc000302e00)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0xc002999a28, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc000302e00, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc000302e00, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc000302e00)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc000e6cfd0)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc001696e80, 0x620faa667a5f8}, {0x1879000, 0xc00059c000}, 0x620faa667a5f8)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc001022fb0, {0xc0022d1020, 0x24}, 0x0)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 4543421 [select]:

net/http.(*persistConn).roundTrip(0xc0006ec000, 0xc0006d6380)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c4600)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c4600, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c4600, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c4600)

	net/http/client.go:725 +0x908

net/http.(*Client).Do(...)

	net/http/client.go:593

github.com/hashicorp/nomad/api.(*Client).doRequest(0xc000429ab0, 0xc001784840)

	github.com/hashicorp/nomad/[email protected]/api.go:787 +0x5c

github.com/hashicorp/nomad/api.(*Client).query(0x14cc2f4, {0xc0010d4840, 0x61f98f2350c7c}, {0x1879000, 0xc0003e0700}, 0x61f98f2350c7c)

	github.com/hashicorp/nomad/[email protected]/api.go:917 +0xb4

github.com/hashicorp/nomad/api.(*Scaling).GetPolicy(0xc00184b7b0, {0xc0022d11a0, 0x24}, 0xc0006e8130)

	github.com/hashicorp/nomad/[email protected]/scaling.go:29 +0x88

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy.func1()

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:198 +0xe9

created by github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:191 +0x367



goroutine 1553945 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc0017d7500}, {{0xc0022d0cc0, 0x24}, 0xc000b31560, 0xc000b316e0, 0xc000b31500})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4504308 [select, 5 minutes]:

net/http.(*persistConn).writeLoop(0xc0005d0a20)

	net/http/transport.go:2386 +0xfb

created by net/http.(*Transport).dialConn

	net/http/transport.go:1748 +0x1e65



goroutine 1553911 [select, 5 minutes]:

github.com/hashicorp/nomad-autoscaler/policy/nomad.(*Source).MonitorPolicy(0xc0008b0400, {0x1e4cfb0, 0xc000b80b80}, {{0xc0022d0c90, 0x24}, 0xc000b31380, 0xc000b314a0, 0xc000b31320})

	github.com/hashicorp/nomad-autoscaler/policy/nomad/source.go:202 +0x3ed

created by github.com/hashicorp/nomad-autoscaler/policy.(*Handler).Run

	github.com/hashicorp/nomad-autoscaler/policy/handler.go:115 +0x2e5



goroutine 4543308 [select, 3 minutes]:

net/http.(*persistConn).roundTrip(0xc002c98240, 0xc002556140)

	net/http/transport.go:2614 +0x97d

net/http.(*Transport).roundTrip(0xc00012e780, 0xc0007c5000)

	net/http/transport.go:594 +0x7d1

net/http.(*Transport).RoundTrip(0x30, 0x1e16f40)

	net/http/roundtrip.go:18 +0x19

net/http.send(0xc0007c5000, {0x1e16f40, 0xc00012e780}, {0x1b77dc0, 0x40d101, 0x0})

	net/http/client.go:252 +0x5d8

net/http.(*Client).send(0xc00010efc0, 0xc0007c5000, {0x203000, 0x203000, 0x0})

	net/http/client.go:176 +0x9b

net/http.(*Client).do(0xc00010efc0, 0xc0007c5000)

	net/http/client.go:725 +0x908

[lgfa29]

Hi @bernardoVale 👋

What kind of policies are these? Are they application scaling defined in jobs?

If that's the case, it seems to be coming from Nomad itself, more specifically from here:
https://github.com/hashicorp/nomad/blob/dd15bdff9c35be3fe2d0c470feea016967acea9d/helper/pool/pool.go#L512

Have you noticed any errors in the Nomad log when this happens?

[bernardoVale]

What kind of policies are these? Are they application scaling defined in jobs?

Correct, using nomad target plugin

Have you noticed any errors in the Nomad log when this happens?

It happened again today, and not, there was no error log in nomad server/clients

[nick-kentik]

We had this recur today, and I thought to dig into the nomad-autoscaler logs to see what was happening at the time the errors started. It looked like:

[...]
2024-07-01T03:09:20.575Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: query="Meta contains \"run_foo-agent\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=10 ineligible=0 job=foo pool=all timestamp=2024-07-01T03:09:20.575Z
2024-07-01T03:09:20.581Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: query="Meta contains \"run_bar-gateway\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=1 ineligible=0 job=bar-gateway pool=default timestamp=2024-07-01T03:09:20.581Z
2024-07-01T03:09:20.590Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: ineligible=0 job=baz pool=default query="Meta contains \"node_type\" and Meta[\"node_type\"] == \"storage\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=3 timestamp=2024-07-01T03:09:20.590Z
2024-07-01T03:10:16.280Z [ERROR] policy_manager.policy_handler: rpc error: code = Unavailable desc = error reading from server: EOF: policy_id=a53e053c-0c60-1296-3e35-8e6ca0aa1a6f
2024-07-01T03:10:16.280Z [ERROR] external_plugin: plugin process exited: plugin=/opt/autoscaler-plugins/kentik-nomad-target id=17 error="exit status 2"
2024-07-01T03:11:17.874Z [ERROR] policy_manager.policy_handler: rpc error: code = Canceled desc = context canceled: policy_id=854225b7-34cd-e54c-7f0f-c06fd8be8903
2024-07-01T03:11:17.874Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=854225b7-34cd-e54c-7f0f-c06fd8be8903 error="rpc error: code = Canceled desc = context canceled"
[...]

So it begins to happen after the external plugin (nomad target) exits with an error. I went looking for the code that restarts the plugin on error, and couldn't find it. I did find this: https://github.com/hashicorp/nomad-autoscaler/blob/main/plugins/manager/manager.go#L202

	// TODO(jrasell) if we do not find the instance, we should probably try and
	//  dispense the plugin. We should also check the plugin instance has not
	//  exited.

So is it the case that nomad-autoscaler simply can't recover if an external plugin dies? The 'rpc error:' text is generic enough that I wonder if perhaps the error message is coming from the gRPC communication between plugin client and server, rather than between plugin server and nomad API?

[nick-kentik]

OK, I just tried this out by manually running kill <pid of kentik-nomad-target process>

The logs were a fine copy of the above:

[...]
2024-07-02T12:43:13.258Z [INFO]  external_plugin.nomad-meta-apm-v2: query response: job=foo pool=default query="Meta contains \"run_foo\" and SchedulingEligibility == \"eligible\" and Attributes contains \"driver.docker\"" total=2 ineligible=0 timestamp=2024-07-02T12:43:13.258Z
2024-07-02T12:44:22.364Z [ERROR] external_plugin: plugin process exited: plugin=/opt/autoscaler-plugins/kentik-nomad-target id=40 error="signal: terminated"
2024-07-02T12:45:10.344Z [WARN]  policy_manager.policy_handler: failed to get target status: policy_id=ff2ef4dc-b2b7-297e-ed89-e91cfd65cace error="rpc error: code = Canceled desc = context canceled"
[...]

So in terms of possible solutions, I guess one of:

• plugins should never die
• nomad-autoscaler should die if a plugin dies
• nomad-autoscaler should restart a plugin if it does

?

[nick-kentik]

I've got a test case and first pass at a fix in #931