Skip to content

spraakbanken/sblex-rs

Repository files navigation

sblex-rs

Memory usage

Morphology is the beast in the room that uses a lot of memory.

Python reference: Diagram showing memory usage of Python version

NOTE All rust examples also serializes the trie to json.

Baseline:

Diagram showing memory usage of load_morphology

Use ArcStr instead of String

We can replace String with arcstr::ArcStr as value in the Trie to reduce memory usage.

Diagram showing memory usage of load_morphology

And if also use ArcStr as keys in the inner map we can reduce it some more:

Diagram showing memory usage of load_morphology

Decomposing builder

The builder already holds the number of states and all states, lets grab them (and reuse the memory).

Diagram showing memory usage of load_morphology

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published