Add DenseMoELayer
and wire it up in Mixtral/Deepseek V2 (#2537)
#1216
Job | Run time |
---|---|
7m 11s | |
8m 35s | |
6m 50s | |
7m 22s | |
0s | |
0s | |
0s | |
46m 14s | |
1h 16m 12s |