How to use openeurollm/tokenizer-256k-v2 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("openeurollm/tokenizer-256k-v2", device_map="auto")
a25c453 c6ca0b9 a25c453
1
2
3
4
version https://git-lfs.github.com/spec/v1 oid sha256:a352df5cda48dd73b30bd94bc15b6ca03fe0ca5944bdea4870d47c790f738c47 size 4578178
