Tokenizer JSON

#17
by rageltman - opened

Most of the Candle ecosystem inferencing engines want to parse a tokenizer.json file - Error: request error: https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct/resolve/main/tokenizer.json: status code 404, is there any chance that one can be generated or do we need to pull in tiktoken libraries to communicate with this derivation?

Sign up or log in to comment