Tokenizer JSON
#17
by
rageltman
- opened
Most of the Candle ecosystem inferencing engines want to parse a tokenizer.json file - Error: request error: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/tokenizer.json: status code 404, is there any chance that one can be generated or do we need to pull in tiktoken libraries to communicate with this derivation?