Post
76
Super impressed with the UltraBERT team at @dmis-lab — @affjljoo3581 , @Taewhoo and Jaewoo Kang — for continuing to push encoders forward. Really impressive work!
I knocked up an MoE version on a single A100, entirely for fun: UltraBERT-Embed-MoE-8x.
8 experts per routed layer, 2 active per token. Works pretty well in my small retrieval tests!
Huge credit to the original team for making this possible.
Original: dmis-lab/UltraBERT-Embed-150M
MoE: takarajordan/UltraBERT-Embed-MoE-8x
I knocked up an MoE version on a single A100, entirely for fun: UltraBERT-Embed-MoE-8x.
8 experts per routed layer, 2 active per token. Works pretty well in my small retrieval tests!
Huge credit to the original team for making this possible.
Original: dmis-lab/UltraBERT-Embed-150M
MoE: takarajordan/UltraBERT-Embed-MoE-8x