[Enhancement]: Refine Delegator L0 segment & delete buffer

[congqixia]

Is there an existing issue for this?

• I have searched the existing issues

What would you like to be added?

Reduce the memory usage for delegator with large number of l0 segments or delete buffer

Why is this needed?

Before streaming node implementation, delegator cannot control the size of l0 segments and delete buffer. In some cases the memory pressure could be too large to handle.
We would like to change the l0 "loading" and split process in delegator and do some refinement for delete buffer purging

Anything else?

No response

[congqixia]

• Load deltalog in worker instead of delegator
  • Store l0 deltalog meta only and dispatch logs when loading segments enhance: Add L0 forward policy to support remote load #36189
  • Utilize PK index when load deltalog files to discard delete entries which does not hit enhance: Filter out non-hit delete records during load delta #36207
  • Add deltalog cache for querynode worker to reduce duplicated object store IO
• Purge delete buffer asap when all l0 segments for current target is ready enhance: Purge delete buffer asap after all L0 segment ready #35304
• Split delete data when streaming delete size is too large
• (optional) Flush delete buffer into oss files when delete buffer is too large, shall be easier to implemented with streaming node

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.