Define the format, metadata for shared probability tables, strings, etc.

[dominiccooney]

The binast-node repo and this repo's entropy branch are experimenting with entropy-based encoding methods. To achieve better file sizes, there's an idea to share probability tables and other data between files in a corpus. We need to define what this format is.

Because a lot of data in this file depends on the grammar, possibly the grammar should be part of this "dictionary" file, or the dictionary file should refer to the grammar.

Because a given file would not be usable with a different dictionary, it probably makes sense for a file to have metadata pointing to its dictionary as opposed to/in addition to out of band metadata such as a HTTP header.

[Yoric]

Discussion moved here.

[Yoric]

Because a given file would not be usable with a different dictionary, it probably makes sense for a file to have metadata pointing to its dictionary as opposed to/in addition to out of band metadata such as a HTTP header.

That specific point moved here.