loki_out: add stuctured_metadata_map_keys #9530
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
0x006EA1E5
requested review from
edsiper,
leonardo-albertovich,
fujimotos and
koleini
as code owners
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
7f1db76 to
a9cdcad
Compare
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
a9cdcad to
e9fee7d
Compare
0x006EA1E5
commented
Nov 28, 2024
| @@ -78,6 +78,7 @@ struct flb_loki { | |||
| struct mk_list *labels; | |||
| struct mk_list *label_keys; | |||
| struct mk_list *structured_metadata; | |||
| struct mk_list *structured_metadata_map_keys; | |||
There was a problem hiding this comment.
is this config property name okay structured_metadata_map_keys?
0x006EA1E5
commented
Nov 28, 2024
0x006EA1E5
commented
Nov 28, 2024
| "{\"sub_key1\": \"sub_value1\", \"sub_key2\": false}}, \"map2\": {\"key4\": " \ | ||
| "\"value1\", \"key5\": false}, \"map3\": {\"key1\": \"map3_value1\", \"key2\": " \ | ||
| "\"map3_value2\"}}]" | ||
| void flb_test_structured_metadata_map_params(char *remove_keys, |
There was a problem hiding this comment.
There are quite a few different variations on this test, so I have created this function which takes various parameters, and then called it for each actual test (below)
0x006EA1E5
commented
Nov 28, 2024
| {"structured_metadata_map_and_explicit", | ||
| flb_test_structured_metadata_map_and_explicit}, | ||
| {"structured_metadata_map_single_missing_map", | ||
| flb_test_structured_metadata_map_single_missing_map}, |
There was a problem hiding this comment.
Any test cases missing that I should add?
0x006EA1E5
commented
Nov 28, 2024
| &ctx->structured_metadata_map_keys_list); | ||
| } | ||
| /* | ||
| * explicit structured_metadata entries override |
There was a problem hiding this comment.
I believe this works as expected.
The intention is, if we try to add an entry to the structured_metadata with the same key twice (first from the new structured_metadata_map_keys, then from the existing structured_metadata), then the second entry should "win" and overrwite the first.
I am assuming msgpack works this way. But maybe I actually need to do some kind of explicit check, and only add a new entry where there is not already one for a given key?
0x006EA1E5
commented
Nov 28, 2024
| * separate item. Non-string map values are serialised to JSON, as Loki requires | ||
| * all values to be strings. | ||
| */ | ||
| static void pack_maps(struct flb_loki *ctx, |
There was a problem hiding this comment.
This essentially tries to follow the flow of the proceeding pack_kv function.
Do you need a more detailed spec of what exactly this is supposed to be doing?
It should implement the proposal descibed in the FR #9463
|
I added a PR for the docs here fluent/fluent-bit-docs#1527 |
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
f3c4f5e to
4f158a5
Compare
|
Example config file, for sending logs to Loki at Grafana Cloud: Debug log output: |
|
valgrind output for |
|
valgrind output from test config (confirmed sending logs correctly downstream) |
|
Screenshot of Grafana Loki, showing the remote logs with structured metadata fields |
|
I have now added a docker compose at n.b., to run this for the branch, we of course need to build the image so that Running this, we can go to the Grafana instance running at http://localhost:3000/ and see the logs with the expected structured_metadata: I have a slightly different / updated fluentbit config for this: service:
log_level: debug
pipeline:
inputs:
- name: dummy
tag: logs
dummy: |
{
"message": "simple log generated",
"logger": "my.logger",
"level": "INFO",
"hostname": "localhost",
"my_map_of_attributes_1": {
"key_1": "hello, world!",
"key_2": "goodbye, world!"
},
"my_map_of_maps_1": {
"root_key": {
"sub_key_1": "hello, world!",
"sub_key_2": "goodbye, world!"
}
}
}
outputs:
- name: loki
match: logs
host: loki
remove_keys: hostname,my_map_of_attributes_1,my_map_of_maps_1
label_keys: $level,$logger
labels: service_name=test
structured_metadata: $hostname
structured_metadata_map_keys: $my_map_of_attributes_1,$my_map_of_maps_1['root_key']
line_format: key_value
drop_single_key: on |
* Adds stuctured_metadata_map_keys config to dynamically populate stuctured_metadata from a map Signed-off-by: Greg Eales <[email protected]>
Add docker-compose to test loki backend Signed-off-by: Greg Eales <[email protected]>
0x006EA1E5
force-pushed
the
loki_out-structured_metadata_map
branch
from
da03432 to
02c8797
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #9463
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
fluent/fluent-bit-docs#1527
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.