Generating semantic tokens for training

[yurtmete]

I had the pretty much the same question in #7

Do you think would it make sense to use w2v-bert to extract the semantic tokens for fine-tuning of the first transformer (text model, semantic modeling)?

I am suggesting w2v-bert as it was used AudioLM to generate semantic tokens from audio.

[gkucsko]

afaik w2v-bert is not public, but ya something like that would be a good start i think