Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: improve slice usage from pooled objects #160

Merged
merged 9 commits into from
Sep 26, 2023
Original file line number Diff line number Diff line change
Expand Up @@ -45,18 +45,18 @@ func HTTPHeadersToStructPb(h http.Header) *structpb.Struct {
return nil
}

func HTTPHeadersToModelpb(h http.Header) []*modelpb.HTTPHeader {
func HTTPHeadersToModelpb(h http.Header, out []*modelpb.HTTPHeader) []*modelpb.HTTPHeader {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem right, we're passing out and also returning out?

Copy link
Member Author

@kruskall kruskall Sep 25, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If out is not big enough we might append to it, so we need to return a slice because they might not match

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this API a bit confusing, I think it'd be best to do one or the other. If we want to accept the out as an argument, then we can pass out *[]*modelpb.HTTPHeader

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this signature looks similar to s = append(s, struct{}), so that's an argument for this pattern. But we'll have to remember to use the return value.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But for conversion purposes like this specific one, I think @marclop 's comment makes sense. It will be clearer.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit conflicted. This method was inspired by the slices package in the stdlib (https://pkg.go.dev/slices)

There are several methods/API there that returns the result (modified slice) despite changing the argument (Compact, Grow, Replace, Clip, etc.). I assume they might in turn have copied the signature of append.

I think this is a common pattern in go ecosystem but I don't feel strongly about it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for chiming in. I was unaware of the widespread apdoption of this pattern in the current Go stdlib. I still think it's clearer and simpler to accept a single argument. If we leave it as is, I would prefer if we documented the behaviour since there are currently no godocs for these functions.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

if len(h) == 0 {
return nil
}
headers := make([]*modelpb.HTTPHeader, 0, len(h))
out = Reslice(out, len(h), modelpb.HTTPHeaderFromVTPool)
i := 0
for k, v := range h {
pbheader := modelpb.HTTPHeaderFromVTPool()
pbheader.Key = k
pbheader.Value = v
headers = append(headers, pbheader)
out[i].Key = k
out[i].Value = v
i++
}
return headers
return out
}

// NormalizeHTTPRequestBody recurses through v, replacing any instance of
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,20 +22,21 @@ import (
"google.golang.org/protobuf/types/known/structpb"
)

func ToKv(m map[string]any) []*modelpb.KeyValue {
nm := normalizeMap(m)
if len(nm) == 0 {
func ToKv(m map[string]any, out []*modelpb.KeyValue) []*modelpb.KeyValue {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

m = normalizeMap(m)
if len(m) == 0 {
return nil
}

kv := []*modelpb.KeyValue{}
out = Reslice(out, len(m), modelpb.KeyValueFromVTPool)

i := 0
for k, v := range m {
value, _ := structpb.NewValue(v)
pbkv := modelpb.KeyValueFromVTPool()
pbkv.Key = k
pbkv.Value = value
kv = append(kv, pbkv)
out[i].Key = k
out[i].Value = value
i++
}

return kv
return out
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
// Licensed to Elasticsearch B.V. under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Elasticsearch B.V. licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package modeldecoderutil

// Reslice increases the slice's capacity, if necessary, to match n.
// If specified, the newFn function is used to create the elements
// to populate the additional space appended to the slice.
func Reslice[Slice ~[]model, model any](slice Slice, n int, newFn func() model) Slice {
if diff := n - cap(slice); diff > 0 {
extra := make([]model, diff)
if newFn != nil {
for i := range extra {
extra[i] = newFn()
}
}
slice = append([]model(slice)[:cap(slice)], extra...)
}
return slice[:n]
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
// Licensed to Elasticsearch B.V. under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Elasticsearch B.V. licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package modeldecoderutil

import (
"testing"

"github.com/elastic/apm-data/model/modelpb"
"github.com/stretchr/testify/assert"
)

func TestReslice(t *testing.T) {
originalSize := 10
s := make([]*modelpb.APMEvent, originalSize)
for i := range s {
s[i] = &modelpb.APMEvent{}
}

downsize := 4
s = Reslice(s, downsize, nil)
for _, e := range s {
assert.NotNil(t, e)
}
assert.Equal(t, downsize, len(s))
assert.Equal(t, originalSize, cap(s))

upsize := 20
s = Reslice(s, upsize, func() *modelpb.APMEvent { return &modelpb.APMEvent{} })
assert.Equal(t, upsize, len(s))
for _, e := range s {
assert.NotNil(t, e)
}
assert.Equal(t, upsize, cap(s))

}
31 changes: 14 additions & 17 deletions input/elasticapm/internal/modeldecoder/rumv3/decoder.go
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ func mapToErrorModel(from *errorEvent, event *modelpb.APMEvent) {
}
}
if len(from.Context.Custom) > 0 {
out.Custom = modeldecoderutil.ToKv(from.Context.Custom)
out.Custom = modeldecoderutil.ToKv(from.Context.Custom, out.Custom)
}
}
if from.Culprit.IsSet() {
Expand Down Expand Up @@ -251,7 +251,7 @@ func mapToErrorModel(from *errorEvent, event *modelpb.APMEvent) {
log.ParamMessage = from.Log.ParamMessage.Val
}
if len(from.Log.Stacktrace) > 0 {
log.Stacktrace = make([]*modelpb.StacktraceFrame, len(from.Log.Stacktrace))
log.Stacktrace = modeldecoderutil.Reslice(log.Stacktrace, len(from.Log.Stacktrace), modelpb.StacktraceFrameFromVTPool)
mapToStracktraceModel(from.Log.Stacktrace, log.Stacktrace)
}
out.Log = log
Expand Down Expand Up @@ -285,17 +285,15 @@ func mapToErrorModel(from *errorEvent, event *modelpb.APMEvent) {

func mapToExceptionModel(from errorException, out *modelpb.Exception) {
if len(from.Attributes) > 0 {
out.Attributes = modeldecoderutil.ToKv(from.Attributes)
out.Attributes = modeldecoderutil.ToKv(from.Attributes, out.Attributes)
}
if from.Code.IsSet() {
out.Code = modeldecoderutil.ExceptionCodeString(from.Code.Val)
}
if len(from.Cause) > 0 {
out.Cause = make([]*modelpb.Exception, len(from.Cause))
out.Cause = modeldecoderutil.Reslice(out.Cause, len(from.Cause), modelpb.ExceptionFromVTPool)
for i := 0; i < len(from.Cause); i++ {
ex := modelpb.ExceptionFromVTPool()
mapToExceptionModel(from.Cause[i], ex)
out.Cause[i] = ex
mapToExceptionModel(from.Cause[i], out.Cause[i])
}
}
if from.Handled.IsSet() {
Expand All @@ -308,7 +306,7 @@ func mapToExceptionModel(from errorException, out *modelpb.Exception) {
out.Module = from.Module.Val
}
if len(from.Stacktrace) > 0 {
out.Stacktrace = make([]*modelpb.StacktraceFrame, len(from.Stacktrace))
out.Stacktrace = modeldecoderutil.Reslice(out.Stacktrace, len(from.Stacktrace), modelpb.StacktraceFrameFromVTPool)
mapToStracktraceModel(from.Stacktrace, out.Stacktrace)
}
if from.Type.IsSet() {
Expand Down Expand Up @@ -450,7 +448,7 @@ func mapToTransactionMetricsetModel(from *transactionMetricset, event *modelpb.A

func mapToResponseModel(from contextResponse, out *modelpb.HTTPResponse) {
if from.Headers.IsSet() {
out.Headers = modeldecoderutil.HTTPHeadersToModelpb(from.Headers.Val)
out.Headers = modeldecoderutil.HTTPHeadersToModelpb(from.Headers.Val, out.Headers)
}
if from.StatusCode.IsSet() {
out.StatusCode = uint32(from.StatusCode.Val)
Expand All @@ -474,10 +472,10 @@ func mapToRequestModel(from contextRequest, out *modelpb.HTTPRequest) {
out.Method = from.Method.Val
}
if len(from.Env) > 0 {
out.Env = modeldecoderutil.ToKv(from.Env)
out.Env = modeldecoderutil.ToKv(from.Env, out.Env)
}
if from.Headers.IsSet() {
out.Headers = modeldecoderutil.HTTPHeadersToModelpb(from.Headers.Val)
out.Headers = modeldecoderutil.HTTPHeadersToModelpb(from.Headers.Val, out.Headers)
}
}

Expand Down Expand Up @@ -677,7 +675,7 @@ func mapToSpanModel(from *span, event *modelpb.APMEvent) {
out.RepresentativeCount = 1 / from.SampleRate.Val
}
if len(from.Stacktrace) > 0 {
out.Stacktrace = make([]*modelpb.StacktraceFrame, len(from.Stacktrace))
out.Stacktrace = modeldecoderutil.Reslice(out.Stacktrace, len(from.Stacktrace), modelpb.StacktraceFrameFromVTPool)
mapToStracktraceModel(from.Stacktrace, out.Stacktrace)
}
if from.Sync.IsSet() {
Expand All @@ -693,7 +691,7 @@ func mapToSpanModel(from *span, event *modelpb.APMEvent) {

func mapToStracktraceModel(from []stacktraceFrame, out []*modelpb.StacktraceFrame) {
for idx, eventFrame := range from {
fr := modelpb.StacktraceFrameFromVTPool()
fr := out[idx]
if eventFrame.AbsPath.IsSet() {
fr.AbsPath = eventFrame.AbsPath.Val
}
Expand Down Expand Up @@ -721,14 +719,13 @@ func mapToStracktraceModel(from []stacktraceFrame, out []*modelpb.StacktraceFram
fr.Module = eventFrame.Module.Val
}
if len(eventFrame.PostContext) > 0 {
fr.PostContext = make([]string, len(eventFrame.PostContext))
fr.PostContext = modeldecoderutil.Reslice(fr.PostContext, len(eventFrame.PostContext), nil)
copy(fr.PostContext, eventFrame.PostContext)
}
if len(eventFrame.PreContext) > 0 {
fr.PreContext = make([]string, len(eventFrame.PreContext))
fr.PreContext = modeldecoderutil.Reslice(fr.PreContext, len(eventFrame.PreContext), nil)
copy(fr.PreContext, eventFrame.PreContext)
}
out[idx] = fr
}
}

Expand Down Expand Up @@ -761,7 +758,7 @@ func mapToTransactionModel(from *transaction, event *modelpb.APMEvent) {
// map transaction specific data
if from.Context.IsSet() {
if len(from.Context.Custom) > 0 {
out.Custom = modeldecoderutil.ToKv(from.Context.Custom)
out.Custom = modeldecoderutil.ToKv(from.Context.Custom, out.Custom)
}
if len(from.Context.Tags) > 0 {
modeldecoderutil.MergeLabels(from.Context.Tags, event)
Expand Down
Loading