Change the IAVL iterators to not be channel based

[ValarDragon]

Summary

cref tendermint/tm-db#188.

The IAVL iterator creates a channel, and spawns multiple processes in order to work at the moment. This is inefficient, as we get lots of time spent in checking mutex locks, inter-channel message passing, and extra processes lying around within the state machine. (It will also be a barrier to parallelism in the future)

This all only makes sense if we're designing for the use case of multiple threads reading from the same iterator, which doesn't make sense. A good parallelism design would aim to make multiple iterators over non-overlapping parts of state. Even if we were designing for this, only the mutex locks make sense, the separately spawned iterator channels do not.

I recommend that we change this to remove the need for the extraneous channels entirely, and just have a struct maintain the context. This is much simpler, and faster to execute. I think we should also remove the locks, but thats secondary to removing the channels.

Problem Definition

IAVL store iterators are resource intensive.

Proposal

• Remove the channels entirely within the iterators, and just use structs there.
• Remove the locks in the iterator as well. (These locks only belong if multiple processes need to read from the same IAVL iterator, which does not make sense to me)

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[ValarDragon]

@mconcat pointed out to me that this is actually the same issue as the b-tree. The IAVL is expecting rust-like, iterators being used in place with a function as an argument syntax.

Whereas the SDK is expecting iterators to just return a value, and does a multi-threading hack thats horrible for performance to make it happen.

The ideal would be to use rust-like iterator syntax throughout the SDK. But in lieue of that massive breaking change (or perhaps a second API), I think we have to update both these underlying db's to support our existing usage

[tac0turtle]

cc @roysc so he is aware of this for the new db package.