CPU is getting exhausted with minimal load on the cluster and KSQL becomes unresponsive

[swathimocharla]

System setup:

Started with one replica and added a batch of streams with existing pods being monitored across multiple tests with varying CPU and number of streams.
KSQL cluster was scaled and monitored.
Basic DELIMITED filter streams were added without any load. Streams varied from 20 to 250
Thread count is the default
CPU was varied from 1 to 4

It is observed that every time a new batch of streams is added, there is a spike noticed in the pod in terms of CPU
If the ksql server is almost at the CPU limit after the stabilization, scaling is not really effective, each new pod that comes up almost hits the CPU limit. Substantial latencies are observed intermittently before and after scaling. Increasing the number replicas does not improve the cluster's current CPU utilization. We see very often that ksql cli is in an error state or an unresponsive state

We also notice a huge number of threads created and most of these threads are in error or blocked state.

Is there a mitigation/ work around to get out of this unresponsive state? Clearly, scaling the cluster at this point is uneffective.