How to get Erebus working with Flexgen?

[Slug-Cat]

I've seen some people around who have apparently gotten Erebus 13B to work with FlexGen. This makes sense, it's the same architecture as the properly supported models, but all I've tried results in errors and failure to load the converted model. Anybody feel like writing a guide or providing modified files for it?

[Mar2ck]

You can install this flexgen fork pip install git+https://github.com/Sumanai/FlexGen.git@erebus and rename the model folders to the format opt-6.7b-erebus and that should work for 6.7b and 2.7b.
For 13B you'll have to go to site-packages/flexgen/opt_config.py and add a new entry. Eg:

elif arch_name == "opt-13b-erebus":
  config = OptConfig(name=name,
    max_seq_len=2048, num_hidden_layers=40, n_head=40,
    hidden_size=5120, input_dim=5120, ffn_embed_dim=5120 * 4,
    vocab_size=50265
  )