You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Subscribing (viewport or full table) to a large sparse table requires serializing the full rowset and appending that to the barrage metadata message. Presently we write the "added" to the FlatBufferBuilder early, then add other rowsets and messages, which can result in a case where we resize (allocating a new ByteBuffer and copying existing data) from 1024 to "very big" early, and then resize several more times as we append more data to send.
In theory we could start off at a better size by measuring how big each vector will be (caveat: this won't work quite the same with viewports). Another option could be to write the added rows as late as possible, so that large sparse snapshots only pay a fraction of the cost.
Fetching this in the browser with a small viewport results in around 100-200ms in processing to copy bytebuffers:
With viewport changes to send minimal updates outside of the visible range, this will only impact full table subscriptions like charts, worker-to-worker.
The text was updated successfully, but these errors were encountered:
Discovered researching #6177.
Subscribing (viewport or full table) to a large sparse table requires serializing the full rowset and appending that to the barrage metadata message. Presently we write the "added" to the FlatBufferBuilder early, then add other rowsets and messages, which can result in a case where we resize (allocating a new ByteBuffer and copying existing data) from 1024 to "very big" early, and then resize several more times as we append more data to send.
In theory we could start off at a better size by measuring how big each vector will be (caveat: this won't work quite the same with viewports). Another option could be to write the added rows as late as possible, so that large sparse snapshots only pay a fraction of the cost.
Example code:
Fetching this in the browser with a small viewport results in around 100-200ms in processing to copy bytebuffers:
With viewport changes to send minimal updates outside of the visible range, this will only impact full table subscriptions like charts, worker-to-worker.
The text was updated successfully, but these errors were encountered: