huggingface/data-measurements-tool at 6af9ef60a8cd18aeb7a113298d4d457ea7942a3f

data-measurements-tool / data_measurements

108 kB

14 contributors

meg-huggingface

Splitting prepare_dataset into preparing the base dataset, and the tokenized dataset. This will help us to have further control over caching and loading data, eventually removing the storage of base dataset.

6af9ef6 about 4 years ago

__init__.py
0 Bytes

:tada: init about 4 years ago
dataset_statistics.py
42.5 kB

Splitting prepare_dataset into preparing the base dataset, and the tokenized dataset. This will help us to have further control over caching and loading data, eventually removing the storage of base dataset. about 4 years ago
dataset_utils.py
9.7 kB

:tada: init about 4 years ago
embeddings.py
16.3 kB

:tada: init about 4 years ago
npmi.py
10.5 kB

:bug: really make sure log_files/ exists about 4 years ago
streamlit_utils.py
20.5 kB

:tada: init about 4 years ago
zipf.py
8.86 kB

:bug: really make sure log_files/ exists about 4 years ago