Morphology is the beast in the room that uses a lot of memory.
NOTE All rust examples also serializes the trie to json.
Baseline:
We can replace String
with arcstr::ArcStr
as value in the Trie
to reduce memory usage.
And if also use ArcStr
as keys in the inner map we can reduce it some more:
The builder already holds the number of states and all states, lets grab them (and reuse the memory).