Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support JSON Lines output format for server req #914

Open
ripienaar opened this issue Oct 26, 2023 · 0 comments
Open

Support JSON Lines output format for server req #914

ripienaar opened this issue Oct 26, 2023 · 0 comments
Labels
proposal Enhancement idea or proposal

Comments

@ripienaar
Copy link
Collaborator

ripienaar commented Oct 26, 2023

Proposed change

Today people reach to nats server report connections --json to get a full fleet view of connections since it:

  • Will fetch all the pages
  • Will flatten them all into one enormous struct which is theoretically easier to use

The problem is 30k connections equal about 30GB of RAM when unmarshalled in order to flatten them - and then that has to be marshalled again. So its almost entirely unusable.

But this is unavoidable as I have to build the final structure.

A better approach is to support https://jsonlines.org/ for these enormous use cases, we'd then turn each conn into a json object with server and conn and emit them one line at a time. Thus we can use almost no RAM, just enough to hold one page of responses in memory and unmarshal one response at a time.

So we should:

  • Add json lines to nats server req *
  • Make a generic paged request helper and support paging in all of nats server req * when JSON lines is enabled
  • Hard cap nats server report to some sane limit like 1000 and steer people to JSON lines support instead

Use case

Reporting on large infra

Contribution

No response

@ripienaar ripienaar added the proposal Enhancement idea or proposal label Oct 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
proposal Enhancement idea or proposal
Projects
None yet
Development

No branches or pull requests

1 participant