Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.co... #137906

Open
cockroach-sentry opened this issue Dec 22, 2024 · 1 comment
Open

Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.co... #137906

cockroach-sentry opened this issue Dec 22, 2024 · 1 comment
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-kv KV Team X-blathers-triaged blathers was able to find an owner

Comments

@cockroach-sentry
Copy link
Collaborator

cockroach-sentry commented Dec 22, 2024 •
edited by cockroach-jira-scripts
Loading

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6169872013/?referrer=webhooks_plugin

Panic Message:

panic.go:884: runtime error: invalid memory address or nil pointer dereference
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:122
  | runtime.gopanic
  | 	GOROOT/src/runtime/panic.go:890
  | runtime.panicmem
  | 	GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  | 	GOROOT/src/runtime/signal_unix.go:835
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).GetSpans
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:225
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).Len
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:138
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset.(*SpanSet).Empty
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/spanset/spanset.go:146
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:301
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:224
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:453
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:206
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1189
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
  | 	github.com/cockroachdb/cockroach/pkg/server/node.go:1273
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:841
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:96
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:276
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:243
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
  | 	github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:322
  | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:241
  | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:952
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:851
  | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
  | 	github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:227
  | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1036
  | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:921
  | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
  | 	github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1044
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:210
  | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext
  | 	github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189
Wraps: (2) runtime error: invalid memory address or nil pointer dereference
Error types: (1) *withstack.withStack (2) runtime.errorString
-- report composition:
runtime.errorString
panic.go:884: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 188 to 190 in cf85e8a

}
return gt.sendBatch(ctx, r.NodeID, iface, ba)
}

cockroach/pkg/kv/kvclient/kvcoord/transport.go

Lines 209 to 211 in cf85e8a

}
reply, err := iface.Batch(ctx, ba)
// If we queried a remote node, perform extra validation.

https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L1043-L1045
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L920-L922
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L1035-L1037

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 226 to 228 in cf85e8a

if localRequest {
return invoker(ctx, method, req, resp, cc, opts...)
}

https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L850-L852
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L240-L242

cockroach/pkg/util/stop/stopper.go

Lines 321 to 323 in cf85e8a

return f(ctx)
}

https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L242-L244
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L951-L953
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L275-L277
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L951-L953

cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go

Lines 95 to 97 in cf85e8a

if methodExcludedFromTracing(info.FullMethod) {
return handler(ctx, req)
}

https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/rpc/pkg/rpc/context.go#L840-L842

cockroach/pkg/server/node.go

Lines 1272 to 1274 in cf85e8a

br, err := n.batchInternal(ctx, tenantID, args)

cockroach/pkg/server/node.go

Lines 1188 to 1190 in cf85e8a

var pErr *kvpb.Error
br, writeBytes, pErr = n.stores.SendWithWriteBytes(ctx, args)
if pErr != nil {

https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/kv/kvserver/pkg/kv/kvserver/stores.go#L202-L204
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L205-L207
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L184-L186
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go#L452-L454

cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go

Lines 223 to 225 in cf85e8a

g.EvalKind = evalKind
resp, err := m.sequenceReqWithGuard(ctx, g, branch)
if resp != nil || err != nil {

cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go

Lines 300 to 302 in cf85e8a

// Some requests don't want the wait on locks.
if g.Req.LockSpans.Empty() {
return nil, nil

cockroach/pkg/kv/kvserver/spanset/spanset.go

Lines 145 to 147 in cf85e8a

func (s *SpanSet) Empty() bool {
return s.Len() == 0
}

cockroach/pkg/kv/kvserver/spanset/spanset.go

Lines 137 to 139 in cf85e8a

for ss := SpanScope(0); ss < NumSpanScope; ss++ {
count += len(s.GetSpans(sa, ss))
}

cockroach/pkg/kv/kvserver/spanset/spanset.go

Lines 224 to 226 in cf85e8a

func (s *SpanSet) GetSpans(access SpanAccess, scope SpanScope) []Span {
return s.spans[access][scope]
}

GOROOT/src/runtime/signal_unix.go#L834-L836
GOROOT/src/runtime/panic.go#L259-L261
GOROOT/src/runtime/panic.go#L889-L891
https://github.com/cockroachdb/cockroach/blob/cf85e8ad48bb7a8afd821863d4f345baafa4a1b6/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go#L121-L123
GOROOT/src/runtime/panic.go#L883-L885

pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext at line 189
pkg/kv/kvclient/kvcoord/transport.go in pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch at line 210
pkg/rpc/pkg/rpc/context.go in pkg/rpc.internalClientAdapter.Batch at line 1044
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func3 at line 921
pkg/rpc/pkg/rpc/context.go in pkg/rpc.getChainUnaryInvoker.func1 at line 1036
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2 at line 227
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func2 at line 851
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1 at line 241
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunTaskWithErr at line 322
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func1.1 at line 243
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/rpc/pkg/rpc/context.go in pkg/rpc.NewServerEx.func3 at line 276
pkg/rpc/pkg/rpc/context.go in pkg/rpc.bindUnaryServerInterceptorToHandler.func1 at line 952
pkg/util/tracing/grpcinterceptor/grpc_interceptor.go in pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1 at line 96
pkg/rpc/pkg/rpc/context.go in pkg/rpc.makeInternalClientAdapter.func1 at line 841
pkg/server/node.go in pkg/server.(*Node).Batch at line 1273
pkg/server/node.go in pkg/server.(*Node).batchInternal at line 1189
pkg/kv/kvserver/pkg/kv/kvserver/stores.go in pkg/kv/kvserver.(*Stores).SendWithWriteBytes at line 203
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes at line 206
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).SendWithWriteBytes at line 185
pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go in pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries at line 453
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq at line 224
pkg/kv/kvserver/concurrency/concurrency_manager.go in pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard at line 301
pkg/kv/kvserver/spanset/spanset.go in pkg/kv/kvserver/spanset.(*SpanSet).Empty at line 146
pkg/kv/kvserver/spanset/spanset.go in pkg/kv/kvserver/spanset.(*SpanSet).Len at line 138
pkg/kv/kvserver/spanset/spanset.go in pkg/kv/kvserver/spanset.(*SpanSet).GetSpans at line 225
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 835
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 890
pkg/kv/kvserver/pkg/kv/kvserver/store_send.go in pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1 at line 122
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag Value
Command server
Environment v23.1.4
Go Version go1.19.4
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.4
Cockroach SHA cf85e8a
# of CPUs 8
# of Goroutines 389

Jira issue: CRDB-45805
@cockroach-sentry cockroach-sentry added branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Dec 22, 2024
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Dec 22, 2024

CC'ing via the CODEOWNERS-based sentry heuristic:

  • @cockroachdb/kv

🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

@blathers-crl blathers-crl bot added T-kv KV Team X-blathers-triaged blathers was able to find an owner labels Dec 22, 2024
@exalate-issue-sync exalate-issue-sync bot changed the title Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.co... Sentry: panic.go:884: runtime error: invalid memory address or nil pointer dereference (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.co... Dec 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
branch-release-23.1 Used to mark GA and release blockers, technical advisories, and bugs for 23.1 C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. T-kv KV Team X-blathers-triaged blathers was able to find an owner
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cockroach-sentry