Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http_json_handler: Don't have err failed to write response log entire error #1

Open
4 tasks
ValarDragon opened this issue Jan 14, 2023 · 1 comment
Open
4 tasks

Comments

@ValarDragon
Copy link

Summary

There are many causes of failed to write response errors in this file: https://github.com/tendermint/tendermint/blob/v0.34.24/rpc/jsonrpc/server/http_json_handler.go#L19-L129

One of these frequently leads to massive "log bombs", that spans thousands of characters and crowds out all content in the CLI logs (and looks quite concerning)

 Nov 17 01:49:48 <node_name> cosmovisor[1541013]: 1:49AM ERR failed to write responses err="write tcp 127.0.0.1:26657->127.0.0.1:34266: i/o timeout" module=rpc-server res=[{"id":998254319713,"jsonrpc":"2.0","result":{"total_count":"329048","txs":$MASSIVE_AMOUNT_OF_TEXT

There are some \n characters in this output that end up breaking into multiple lines as well, and breaking parsing formats.

Can we make a smaller result serialization here, that just has a truncated / trimmed amount of data, that gets output to the CLI?

This would significantly reduce the size of logs, and the amount of information that can be easily gleaned by scrolling through them.


For Admin Use

  • Not duplicate issue
  • Appropriate labels applied
  • Appropriate contributors tagged
  • Contributor assigned/self-assigned
@lasarojc
Copy link

Thank you for opening this issue. As previously communicated (https://twitter.com/informalinc/status/1613580979813191683), we won't be making changes in this fork except for dealing with critical bugs/security issues, but we are watching your contributions and will consider them in our rebranded fork, soon to be announced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants