Support multiple GGUF files

[EricLBuehler]

Support multiple GGUF files by refactoring the GGUF Content usage.

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 Dockerfile              1           34           25            0            9
 Happy                   1          442          369            0           73
 JSON                    9           21           21            0            0
 Python                 24          864          731           25          108
 TOML                   15          403          365            1           37
-------------------------------------------------------------------------------
 Jupyter Notebooks       1            0            0            0            0
 |- Markdown             1           60           30           22            8
 |- Python               1           96           87            1            8
 (Total)                            156          117           23           16
-------------------------------------------------------------------------------
 Markdown               16         1056            0          782          274
 |- BASH                 6          203          190            0           13
 |- Python               6          121          110            0           11
 |- Rust                 3          185          172            9            4
 (Total)                           1565          472          791          302
-------------------------------------------------------------------------------
 Rust                   93        29569        26925          433         2211
 |- Markdown            47          468            0          455           13
 (Total)                          30037        26925          888         2224
===============================================================================
 Total                 161        32389        28436         1241         2712
===============================================================================
  

[polarathene]

Oh no I was a bit too slow getting my PR ready 💀

I'm not used to contributing on a project where the change I work on is so prone to conflicts from other activity 😓 Should I have a draft PR that I iterate on instead to better communicate this?

[polarathene]

Do you have an example GGUF file where this is actually the case? Or was this a misunderstanding from referenced motivation in #380 ?

[EricLBuehler]

Yes: https://huggingface.co/mradermacher/dolphin-2.9-mixtral-8x22b-GGUF/tree/main

[polarathene]

I don't understand why the sharding is necessary other than as this comment suggests, to workaround size restrictions on a file host?

Seems like it'd be more appropriate to have a tool that concatenates the files together if there's a UX issue you want to address, but I don't really see the point in adding additional complexity / maintenance to support something that should realistically be addressed outside of mistral.rs (runtime) 🤷‍♂️

Similar to the tokenizer patching you added recently, these are workarounds that seem more appropriate as CLI tools to apply "fixes" 🤔

[EricLBuehler]

I'm not used to contributing on a project where the change I work on is so prone to conflicts from other activity 😓 Should I have a draft PR that I iterate on instead to better communicate this?

Would that be for developing a new feature? If so, that sounds good. Otherwise, we have a Discord for this purpose :)

[polarathene]

Would that be for developing a new feature?

No, I was just refactoring on the GGUF tokenizer, but this PR touched quite a bit that I know rebasing my work won't be something I'm interested in doing.

Otherwise, we have a Discord for this purpose :)

I think you saw my github comment on candle about what I was doing before this PR was merged, you had given it a 👍

Good to know though, my mistake.

[EricLBuehler]

Ah sorry, I thought that was a different refactor. I will roll back these changes, as it would be great to see your changes.

Generally, multiple GGUF files is a rare occurrence. Would your PR include support for those?

[polarathene]

Would your PR include support for those?

No, it was just focused on tidying up the GGUF metadata tokenizer file you already put together, and would then have had the from_gguf() model methods also leverage that to tidy that up a little.

On Linux, from what I've read the file is split in parts that you can just run something like:

cat goliath-120b.Q6_K.gguf-split-* > goliath-120b.Q6_K.gguf

A small CLI tool could do similar to glob the related files and merge the output. I don't know for sure but I assume users would be fine with piecing it back into a single file.

I haven't spent much time reading over your PR changes for the feature, so I could be mistaken on your integration to workaround multi-part files, but at a glance it seemed to interleave a bit (multiple loops and other additions to carry support).

Since I didn't implement my PR on the model files, the conflicting files aren't much. I just don't have the energy atm to compare changes I'd need to do.

[EricLBuehler]

Nice! Perhaps we can add such a tool. This PR was a bit hackey, so I actually just rolled back this PR. There is just 1 (easy) conflict now in #389.