Replace data URL in big_patent dataset and support streaming

I first uploaded the data files to the Hub: I think it is a good option because we have git lfs to track versions and changes. Moreover people will be able to make PRs to propose updates on the data files.
- I would have preferred to upload it it to the "data" org namespace, but it is already taken (although not used): might be possible to take it?

As an alternative (and to be consistent with previous datasets), I also uploaded the data files to our AWS bucket.

We should decide which to use (now and for future datasets) and set it here before merging. We should remove the data files for the non-chosen option.

CC: @lhoestq @mariosasko @polinaeterna

Would it make sense to make the dataset a community one (so, create an organization for it) and store the script and the data in a single repository? Just as it is for most of the datasets. That way we can also access the data using a relative path inside the repo (that's not the point though). The point is that to me it seems a bit more straightforward to store everything in one place.

I guess the strong argument against this logic is that in this case the canonical version won't work... But maybe there is some redirecting mechanism I don't know about? :)

Anyway, I'm in favor of hosting data on the Hub instead of AWS :)

I also think storing everything in one place/single repository is the best option.

@polinaeterna Canonical datasets also support data files (see the [`red_caps` repo](https://huggingface.co/datasets/red_caps/tree/main) for instance)

Thanks @polinaeterna and @mariosasko for your comments.

Yes, definitely it is much better to have everything in the same repo.

I'm transferring their data files to the Hub under "big_patent" and deleting them from the other repo and AWS.

This PR replaces the Google Drive URL with our Hub one, once the data owners have approved to host their data on the Hub.

Moreover, this PR makes the dataset streamable.

Fix #4217.


How to load VERY LARGE dataset?

bug

Something isn't working

The `Trainer` support `IterableDataset`, not just datasets.

### System Info

```shell
I am using transformer trainer while meeting the issue.
The trainer requests torch.utils.data.Dataset as input, which loads the whole dataset into the memory at once. Therefore, when the dataset is too large to load, there's nothing I can do except using IterDataset, which loads samples of data seperately, and results in low efficiency.
I wonder if there are any tricks like Sharding in huggingface trainer.
Looking forward to your reply.
```

### Who can help?

Trainer: @sgugger

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

None

### Expected behavior

```shell
I wonder if there are any tricks like fairseq Sharding very large datasets https://fairseq.readthedocs.io/en/latest/getting_started.html.
Thanks a lot!
```


Autoeval config

_The documentation is not available anymore as the PR was closed or merged._

Related to: https://github.com/huggingface/autonlp-backend/issues/414 and https://github.com/huggingface/autonlp-backend/issues/424

The tests are failing due to the changed metadata:

```
got an unexpected keyword argument 'train-eval-index'
```

I think you can fix this by updating the `DatasetMetadata` class and implementing an appropriate `validate_train_eval_index()` function

@lhoestq we are working with an arbitrary set of tags for `autoeval config`. See https://github.com/huggingface/autonlp-backend/issues/414
I need to add a validator function though for the tests to pass. Our set is not well-defined as in the rest https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources. What's a workaround for this?

On the question of validating the `train-eval-index` metadata, I think the simplest approach would be to validate that the required fields exist and not worry about their values (which are open-ended).

For me, the required fields include:

* `config`
* `task`
* `task_id`
* `splits` (train / validation / eval)
* `col_mapping`
* `metrics` (checking that each one has `type`, `name`)

Here I'm using the spec defined in https://github.com/huggingface/autonlp-backend/issues/414 as a guide.

WDYT @lhoestq ?

Makes sense ! Currently the metadata type validator doesn't support subfields - let me open a PR to add it

I ended up improving the metadata validation in this PR x)

In particular:
- I added support YAML keys with dashes instead of underscores for `train-eval-index`
- I added `train-eval-index` validation  with `validate_train_eval_index`. It does nothing fancy, it just checks that it is a list if it exists in the YAML, but feel free to improve it if you want

Let me know if it sounds good to you ! I think we can improve `validate_train_eval_index` in another PR

Come on windows... I didn't do anything advanced...

Anyway, will try to fix this when I get back home x)

> Come on windows... I didn't do anything advanced...
>
> Anyway, will try to fix this when I get back home x)

Hehe, thanks!

> Did I just fix it for windows and now it fails on linux ? xD

Looks like the Heisenberg uncertainty principle is at play here - you cannot simultaneously have unit tests passing in both Linux and Windows 😅

The worst is that the tests pass locally both on my windows and my linux x)

Ok fixed it, the issue came from python 3.6 that doesn't return the right `__origin__` for Dict and List types

> Alright thanks for adding the first Autoeval config ! :D

Woohoo! Thank you so much 🤗

Added autoeval config to imdb as pilot


Autoeval

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4233). All of your documentation changes will be reflected on that endpoint.


adding new tag to tasks.json and modified for existing datasets

closing in favor of https://github.com/huggingface/datasets/pull/4244


Fix invalid url to CC-Aligned dataset

The CC-Aligned dataset url has changed to  https://data.statmt.org/cc-aligned/, the old address http://www.statmt.org/cc-aligned/ is no longer valid


Why the `conll2003` dataset on huggingface only contains the `en` subset? Where is the  German data?

enhancement

New feature or request

Thanks for reporting @beyondguo.

Indeed, we generate this dataset from this raw data file URL: https://data.deepai.org/conll2003.zip
And that URL only contains the English version.

The German data requires payment

The [original task page](https://www.clips.uantwerpen.be/conll2003/ner/) states "The German data is a collection of articles from the Frankfurter Rundschau. The named entities have been annotated by people of the University of Antwerp. Only the annotations are available here. In order to build these data sets you need access to the ECI Multilingual Text Corpus. It can be ordered from the Linguistic Data Consortium (2003 non-member price: US$ 35.00)."

Inflation since 2003 has also affected LDC's prices, and today the dataset [LDC94T5](https://catalog.ldc.upenn.edu/LDC94T5) is available under license for $75 a copy. The [license](https://catalog.ldc.upenn.edu/license/eci-slash-mci-user-agreement.pdf) includes a non-distribution condition, which is probably why the data has not turned up openly.

The ACL hold copyright of this data; I'll mail them and anyone I can find at ECI to see if they'll open this up now. After all, it worked with Microsoft 3DMM, why not here too, after 28 years? :)

![image](https://user-images.githubusercontent.com/37113676/165416606-96b5db18-b16c-4b6b-928c-de8620fd943e.png)

But on huggingface datasets:
![image](https://user-images.githubusercontent.com/37113676/165416649-8fd77980-ca0d-43f0-935e-f398ba8323a4.png)

Where is the  German data?


new task tag

multi-input-text-classification tag for classification datasets that take more than one input


new task tag


Add f1 metric card, update docstring in py file

_The documentation is not available anymore as the PR was closed or merged._


Add pearsonr mc, update functionality to match the original docs

- adds pearsonr metric card
- adds ability to return p-value
    - p-value was mentioned in the original docs as a return value, but there was no option to return it. I updated the _compute function slightly to have an option to return the p-value.


autoeval config

add train eval index for autoeval


Add Accuracy Metric Card

- adds accuracy metric card
- updates docstring in accuracy.py
- adds .json file with metric card and docstring information


Fix description links in dataset cards

Non passing tests are due to other pre-existing errors in dataset cards: not related to this PR.

I noticed many links were not properly displayed (only text, no link) on the Hub because of wrong syntax, e.g.: https://huggingface.co/datasets/big_patent

This PR fixes all description links in dataset cards.


Dictionary Feature

question

Further information is requested

Hi @jordiae,

Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
```python
"list_of_dict_feature": [
    {
        "key1_in_dict": datasets.Value("string"),
        "key2_in_dict": datasets.Value("int32"),
        ...
    }
],
```

Feel free to re-open this issue if that does not work for your use case.

> Hi @jordiae,
>
> Instead of the `Sequence` feature, you can use just a regular list: put the dict between `[` and `]`:
>
> ```python
> "list_of_dict_feature": [
>     {
>         "key1_in_dict": datasets.Value("string"),
>         "key2_in_dict": datasets.Value("int32"),
>         ...
>     }
> ],
> ```
>
> Feel free to re-open this issue if that does not work for your use case.

Thank you

Hi, I'm trying to create the loading script for a dataset in which one feature is a list of dictionaries, which afaik doesn't fit very well the values and structures supported by Value and Sequence. Is there any suggested workaround, am I missing something?

Thank you in advance.


Altered faiss installation comment

Hi ! Can you explain why this change is needed ?

Facebook recommends installing FAISS using conda (https://github.com/facebookresearch/faiss/blob/main/INSTALL.md). pip does not seem to have the latest version of FAISS. The latest version of faiss is 1.7.2 (https://anaconda.org/conda-forge/faiss), but the latest one available through pip is 1.5.3 (https://pypi.org/project/faiss/).


Add F1 Metric Card


Make code for image downloading from image urls cacheable

Fix #4199


Big_Patent dataset broken

Thanks for reporting. The issue seems not to be directly related to the dataset viewer or the `datasets` library, but instead to it being hosted on Google Drive.

See related issues: https://github.com/huggingface/datasets/issues?q=is%3Aissue+is%3Aopen+drive.google.com

To quote [@lhoestq](https://github.com/huggingface/datasets/issues/4075#issuecomment-1087362551):

> PS: if possible, please try to not use Google Drive links in your dataset script, since Google Drive has download quotas and is not always reliable.

We should find out if the dataset license allows redistribution and contact the data owners to propose them to host their data on our Hub.

The data owners have agreed on hosting their data on the Hub.

## Dataset viewer issue for '*big_patent*'

**Link:** *[link to the dataset viewer page](https://huggingface.co/datasets/big_patent/viewer/all/train)*

*Unable to view because it says FileNotFound, also cannot download it through the python API*

Am I the one who added this dataset ? No


Avoid recursion error in map if example is returned as dict value

I noticed this bug while answering [this question](https://discuss.huggingface.co/t/correct-way-to-create-a-dataset-from-a-csv-file/15686/11?u=mariosasko).

This code replicates the bug:
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": ex})
```
and this is the fix for it (before this PR):
```python
from datasets import Dataset
dset = Dataset.from_dict({"en": ["aa", "bb"], "fr": ["cc", "dd"]})
dset.map(lambda ex: {"translation": dict(ex)})
```

Internally, this can be fixed by merging two dicts via dict unpacking (instead of `dict.update) `in `Dataset.map`, which avoids creating recursive dictionaries.

P.S. `{**a, **b}` is slightly more performant than `a.update(b)` in my bencmarks.


Add `drop_last_batch` to `IterableDataset.map`

Addresses this comment: https://github.com/huggingface/datasets/pull/3801#pullrequestreview-901736921


Skip checksum computation in Imagefolder by default

Avoids having to set `ignore_verifications=True` in `load_dataset("imagefolder", ...)` to skip checksum verification and speed up loading.

The user can still pass `DownloadConfig(record_checksums=True)` to not skip this part.


ETT time series dataset

Ready for review.


[Common Voice] Make sure bytes are correctly deleted if `path` exists

cool that you noticed that we store unnecessary bytes again :D

`path` should be set to local path inside audio feature if exist so that bytes can correctly be deleted.


DatasetDict containing Datasets with different features when pushed to hub gets remapped features

bug

Something isn't working

Hi @pietrolesci, thanks for reporting.

Please note that this is a design purpose: a `DatasetDict` has the same features for all its datasets. Normally, a `DatasetDict` is composed of several sub-datasets each corresponding to a different **split**.

To handle sub-datasets with different features, we use another approach: use different **configurations** instead of **splits**.

However, for the moment `push_to_hub` does not support specifying different configurations. IMHO, we should implement this.

Hi @albertvillanova,

Thanks a lot for your reply! I got it now. The strange thing for me was to have it correctly working (i.e., DatasetDict with different features in some datasets) locally and not on the Hub. It would be great to have configuration supported by `push_to_hub`. Personally, this latter functionality allowed me to iterate rather quickly on dataset curation.

Again, thanks for your time @albertvillanova!

Best,
Pietro

Hi! Yes, we should override `DatasetDict.__setitem__` and throw an error if features dictionaries are different. `DatasetDict` is a subclass of `dict`, so `DatasetDict.{update/setdefault}` need to be overridden as well. We could avoid this by subclassing `UserDict`, but then we would get the name collision - `DatasetDict.data` vs. `UserDict.data`. This makes me think we should rename the `data` attribute of `DatasetDict`/`Dataset` for easier dict subclassing (would also simplify https://github.com/huggingface/datasets/pull/3997) and to follow good Python practices. Another option is to have a custom `UserDict` class in `py_utils`, but it can be hard to keep this class consistent with the built-in `UserDict`.

@albertvillanova @lhoestq wdyt?

I would keep things simple and keep subclassing dict. Regarding the features check, I guess this can be done only for `push_to_hub` right ? It is the only function right now that requires the underlying datasets to be splits (e.g. train/test) and have the same features.

Note that later you will be able to push datasets with different features as different dataset **configurations** (similarly to the [GLUE subsets](https://huggingface.co/datasets/glue) for example). We will work on this soon

Hi there,

I am trying to load a dataset to the Hub. This dataset is a `DatasetDict` composed of various splits. Some splits have a different `Feature` mapping. Locally, the DatasetDict preserves the individual features but if I `push_to_hub` and then `load_dataset`, the features are all the same.

Dataset and code to reproduce available [here](https://huggingface.co/datasets/pietrolesci/robust_nli).

In short:

I have 3 feature mapping
```python
Tri_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"]),
    }
)

Ent_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-entailment", "entailment"]),
    }
)

Con_features = Features(
    {
        "idx": Value(dtype="int64"),
        "premise": Value(dtype="string"),
        "hypothesis": Value(dtype="string"),
        "label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"]),
    }
)
```

Then I create different datasets

```python
dataset_splits = {}

for split in df["split"].unique():
    print(split)
    df_split = df.loc[df["split"] == split].copy()

    if split in Tri_dataset:
        df_split["label"] = df_split["label"].map({"entailment": 0, "neutral": 1, "contradiction": 2})
        ds = Dataset.from_pandas(df_split, features=Tri_features)

    elif split in Ent_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-entailment": 0, "entailment": 1})
        ds = Dataset.from_pandas(df_split, features=Ent_features)

    elif split in Con_bin_dataset:
        df_split["label"] = df_split["label"].map({"non-contradiction": 0, "contradiction": 1})
        ds = Dataset.from_pandas(df_split, features=Con_features)

    else:
        print("ERROR:", split)
    dataset_splits[split] = ds
datasets = DatasetDict(dataset_splits)
```

I then push to hub

```python
datasets.push_to_hub("pietrolesci/robust_nli", token="<token>")
```

Finally, I load it from the hub

```python
datasets_loaded_from_hub = load_dataset("pietrolesci/robust_nli")
```

And I get that

```python
datasets["LI_TS"].features != datasets_loaded_from_hub["LI_TS"].features
```

since

```python
"label": ClassLabel(num_classes=2, names=["non-contradiction", "contradiction"])
```

gets remapped to

```python
 "label": ClassLabel(num_classes=3, names=["entailment", "neutral", "contradiction"])
```


Something isn't working

I agree packaged loaders should support `ClassLabel` feature without throwing an error.

@albertvillanova @mariosasko thank you, with that change now I get
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-9-eeb68eeb9bec>](https://localhost:8080/#) in <module>()
     11 )
     12 # You can make this part faster with num_proc=<some int>
---> 13 sentences = sentences.map(lambda ex: features["label"].str2int(ex["label"]) if ex["label"] is not None else None, features=features)
     14 sentences = sentences.shuffle()

8 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
   2193             if processed_inputs is not None and not isinstance(processed_inputs, (Mapping, pa.Table)):
   2194                 raise TypeError(
-> 2195                     f"Provided `function` which is applied to all elements of table returns a variable of type {type(processed_inputs)}. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects."
   2196                 )
   2197             elif isinstance(indices, list) and isinstance(processed_inputs, Mapping):

TypeError: Provided `function` which is applied to all elements of table returns a variable of type <class 'int'>. Make sure provided `function` returns a variable of type `dict` (or a pyarrow table) to update the dataset or `None` if you are only interested in side effects.
```

the error is raised by [this](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_dataset.py#L2221)

```
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in validate_function_output(processed_inputs, indices)
```

@mariosasko changed it like

```python
sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
```

to avoid the above errorr.

Any update on this? Is this correct ?
> @mariosasko changed it like
>
> ```python
> sentences = sentences.map(lambda ex: {"label" : features["label"].str2int(ex["label"]) if ex["label"] is not None else None}, features=features)
> ```
>
> to avoid the above errorr.

### System Info

```shell
- `transformers` version: 4.18.0
- Platform: Linux-5.4.144+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.10.0+cu111 (True)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@LysandreJik

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from datasets import load_dataset,Features,Value,ClassLabel

class_names = ["cmn","deu","rus","fra","eng","jpn","spa","ita","kor","vie","nld","epo","por","tur","heb","hun","ell","ind","ara","arz","fin","bul","yue","swe","ukr","bel","que","ces","swh","nno","wuu","nob","zsm","est","kat","pol","lat","urd","sqi","isl","fry","afr","ron","fao","san","bre","tat","yid","uig","uzb","srp","qya","dan","pes","slk","eus","cycl","acm","tgl","lvs","kaz","hye","hin","lit","ben","cat","bos","hrv","tha","orv","cha","mon","lzh","scn","gle","mkd","slv","frm","glg","vol","ain","jbo","tok","ina","nds","mal","tlh","roh","ltz","oss","ido","gla","mlt","sco","ast","jav","oci","ile","ota","xal","tel","sjn","nov","khm","tpi","ang","aze","tgk","tuk","chv","hsb","dsb","bod","sme","cym","mri","ksh","kmr","ewe","kab","ber","tpw","udm","lld","pms","lad","grn","mlg","xho","pnb","grc","hat","lao","npi","cor","nah","avk","mar","guj","pan","kir","myv","prg","sux","crs","ckt","bak","zlm","hil","cbk","chr","nav","lkt","enm","arq","lin","abk","pcd","rom","gsw","tam","zul","awa","wln","amh","bar","hbo","mhr","bho","mrj","ckb","osx","pfl","mgm","sna","mah","hau","kan","nog","sin","glv","dng","kal","liv","vro","apc","jdt","fur","che","haw","yor","crh","pdc","ppl","kin","shs","mnw","tet","sah","kum","ngt","nya","pus","hif","mya","moh","wol","tir","ton","lzz","oar","lug","brx","non","mww","hak","nlv","ngu","bua","aym","vec","ibo","tkl","bam","kha","ceb","lou","fuc","smo","gag","lfn","arg","umb","tyv","kjh","oji","cyo","urh","kzj","pam","srd","lmo","swg","mdf","gil","snd","tso","sot","zza","tsn","pau","som","egl","ady","asm","ori","dtp","cho","max","kam","niu","sag","ilo","kaa","fuv","nch","hoc","iba","gbm","sun","war","mvv","pap","ary","kxi","csb","pag","cos","rif","kek","krc","aii","ban","ssw","tvl","mfe","tah","bvy","bcl","hnj","nau","nst","afb","quc","min","tmw","mad","bjn","mai","cjy","got","hsn","gan","tzl","dws","ldn","afh","sgs","krl","vep","rue","tly","mic","ext","izh","sma","jam","cmo","mwl","kpv","koi","bis","ike","run","evn","ryu","mnc","aoz","otk","kas","aln","akl","yua","shy","fkv","gos","fij","thv","zgh","gcf","cay","xmf","tig","div","lij","rap","hrx","cpi","tts","gaa","tmr","iii","ltg","bzt","syc","emx","gom","chg","osp","stq","frr","fro","nys","toi","new","phn","jpa","rel","drt","chn","pli","laa","bal","hdn","hax","mik","ajp","xqa","pal","crk","mni","lut","ayl","ood","sdh","ofs","nus","kiu","diq","qxq","alt","bfz","klj","mus","srn","guc","lim","zea","shi","mnr","bom","sat","szl"]
features = Features({ 'label': ClassLabel(names=class_names), 'text': Value('string')})
num_labels = features['label'].num_classes
data_files = { "train": "train.csv", "test": "test.csv" }
sentences = load_dataset("loretoparisi/tatoeba-sentences",
                             data_files=data_files,
                             delimiter='\t',
                             column_names=['label', 'text'],
                             features = features
```

ERROR:
```
ClassLabel(num_classes=403, names=['cmn', 'deu', 'rus', 'fra', 'eng', 'jpn', 'spa', 'ita', 'kor', 'vie', 'nld', 'epo', 'por', 'tur', 'heb', 'hun', 'ell', 'ind', 'ara', 'arz', 'fin', 'bul', 'yue', 'swe', 'ukr', 'bel', 'que', 'ces', 'swh', 'nno', 'wuu', 'nob', 'zsm', 'est', 'kat', 'pol', 'lat', 'urd', 'sqi', 'isl', 'fry', 'afr', 'ron', 'fao', 'san', 'bre', 'tat', 'yid', 'uig', 'uzb', 'srp', 'qya', 'dan', 'pes', 'slk', 'eus', 'cycl', 'acm', 'tgl', 'lvs', 'kaz', 'hye', 'hin', 'lit', 'ben', 'cat', 'bos', 'hrv', 'tha', 'orv', 'cha', 'mon', 'lzh', 'scn', 'gle', 'mkd', 'slv', 'frm', 'glg', 'vol', 'ain', 'jbo', 'tok', 'ina', 'nds', 'mal', 'tlh', 'roh', 'ltz', 'oss', 'ido', 'gla', 'mlt', 'sco', 'ast', 'jav', 'oci', 'ile', 'ota', 'xal', 'tel', 'sjn', 'nov', 'khm', 'tpi', 'ang', 'aze', 'tgk', 'tuk', 'chv', 'hsb', 'dsb', 'bod', 'sme', 'cym', 'mri', 'ksh', 'kmr', 'ewe', 'kab', 'ber', 'tpw', 'udm', 'lld', 'pms', 'lad', 'grn', 'mlg', 'xho', 'pnb', 'grc', 'hat', 'lao', 'npi', 'cor', 'nah', 'avk', 'mar', 'guj', 'pan', 'kir', 'myv', 'prg', 'sux', 'crs', 'ckt', 'bak', 'zlm', 'hil', 'cbk', 'chr', 'nav', 'lkt', 'enm', 'arq', 'lin', 'abk', 'pcd', 'rom', 'gsw', 'tam', 'zul', 'awa', 'wln', 'amh', 'bar', 'hbo', 'mhr', 'bho', 'mrj', 'ckb', 'osx', 'pfl', 'mgm', 'sna', 'mah', 'hau', 'kan', 'nog', 'sin', 'glv', 'dng', 'kal', 'liv', 'vro', 'apc', 'jdt', 'fur', 'che', 'haw', 'yor', 'crh', 'pdc', 'ppl', 'kin', 'shs', 'mnw', 'tet', 'sah', 'kum', 'ngt', 'nya', 'pus', 'hif', 'mya', 'moh', 'wol', 'tir', 'ton', 'lzz', 'oar', 'lug', 'brx', 'non', 'mww', 'hak', 'nlv', 'ngu', 'bua', 'aym', 'vec', 'ibo', 'tkl', 'bam', 'kha', 'ceb', 'lou', 'fuc', 'smo', 'gag', 'lfn', 'arg', 'umb', 'tyv', 'kjh', 'oji', 'cyo', 'urh', 'kzj', 'pam', 'srd', 'lmo', 'swg', 'mdf', 'gil', 'snd', 'tso', 'sot', 'zza', 'tsn', 'pau', 'som', 'egl', 'ady', 'asm', 'ori', 'dtp', 'cho', 'max', 'kam', 'niu', 'sag', 'ilo', 'kaa', 'fuv', 'nch', 'hoc', 'iba', 'gbm', 'sun', 'war', 'mvv', 'pap', 'ary', 'kxi', 'csb', 'pag', 'cos', 'rif', 'kek', 'krc', 'aii', 'ban', 'ssw', 'tvl', 'mfe', 'tah', 'bvy', 'bcl', 'hnj', 'nau', 'nst', 'afb', 'quc', 'min', 'tmw', 'mad', 'bjn', 'mai', 'cjy', 'got', 'hsn', 'gan', 'tzl', 'dws', 'ldn', 'afh', 'sgs', 'krl', 'vep', 'rue', 'tly', 'mic', 'ext', 'izh', 'sma', 'jam', 'cmo', 'mwl', 'kpv', 'koi', 'bis', 'ike', 'run', 'evn', 'ryu', 'mnc', 'aoz', 'otk', 'kas', 'aln', 'akl', 'yua', 'shy', 'fkv', 'gos', 'fij', 'thv', 'zgh', 'gcf', 'cay', 'xmf', 'tig', 'div', 'lij', 'rap', 'hrx', 'cpi', 'tts', 'gaa', 'tmr', 'iii', 'ltg', 'bzt', 'syc', 'emx', 'gom', 'chg', 'osp', 'stq', 'frr', 'fro', 'nys', 'toi', 'new', 'phn', 'jpa', 'rel', 'drt', 'chn', 'pli', 'laa', 'bal', 'hdn', 'hax', 'mik', 'ajp', 'xqa', 'pal', 'crk', 'mni', 'lut', 'ayl', 'ood', 'sdh', 'ofs', 'nus', 'kiu', 'diq', 'qxq', 'alt', 'bfz', 'klj', 'mus', 'srn', 'guc', 'lim', 'zea', 'shi', 'mnr', 'bom', 'sat', 'szl'], id=None)
Value(dtype='string', id=None)
Using custom data configuration loretoparisi--tatoeba-sentences-7b2c5e991f398f39
Downloading and preparing dataset csv/loretoparisi--tatoeba-sentences to /root/.cache/huggingface/datasets/csv/loretoparisi--tatoeba-sentences-7b2c5e991f398f39/0.0.0/433e0ccc46f9880962cc2b12065189766fbb2bee57a221866138fb9203c83519...
Downloading data files: 100%
2/2 [00:18<00:00, 8.06s/it]
Downloading data: 100%
391M/391M [00:13<00:00, 35.3MB/s]
Downloading data: 100%
92.4M/92.4M [00:02<00:00, 36.5MB/s]
Failed to read file '/root/.cache/huggingface/datasets/downloads/933132df9905194ea9faeb30cabca8c49318795612f6495fcb941a290191dd5d' with error <class 'ValueError'>: invalid literal for int() with base 10: 'cmn'
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

TypeError: Cannot cast array data from dtype('O') to dtype('int64') according to the rule 'safe'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
15 frames
/usr/local/lib/python3.7/dist-packages/pandas/_libs/parsers.pyx in pandas._libs.parsers.TextReader._convert_tokens()

ValueError: invalid literal for int() with base 10: 'cmn'
```

while loading without `features` it loads without errors

```
sentences = load_dataset("loretoparisi/tatoeba-sentences",
                             data_files=data_files,
                             delimiter='\t',
                             column_names=['label', 'text']
                         )
```

but the `label` col seems to be wrong (without the `ClassLabel` object):

```
sentences['train'].features
{'label': Value(dtype='string', id=None),
 'text': Value(dtype='string', id=None)}
```

The dataset was https://huggingface.co/datasets/loretoparisi/tatoeba-sentences

Dataset format is:

```
ces	Nechci vědět, co je tam uvnitř.
ces	Kdo o tom chce slyšet?
deu	Tom sagte, er fühle sich nicht wohl.
ber	Mel-iyi-d anida-t tura ?
hun	Gondom lesz rá rögtön.
ber	Mel-iyi-d anida-tt tura ?
deu	Ich will dich nicht reden hören.
```

### Expected behavior

```shell
correctly load train and test files.
```


Add CMU MoCap Dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4208). All of your documentation changes will be reflected on that endpoint.

- Updated the readme.
- Added dummy_data.zip and ran the all the tests.

The dataset works for "asf/amc" and "avi" formats which have a single download link for the complete dataset. But "c3d" and "mpg" have multiple download links, can we combine and host these links on the Hub since the dataset is free to use ?

"c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Can we combine and host these links on the Hub since the dataset is free to use ?

> "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?

We store downloaded data under `~/.cache/huggingface/datasets/downloads` (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".

> We store downloaded data under ~/.cache/huggingface/datasets/downloads (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".

Yes, the filesystem won't be clustered, but the problem is processing the dataset becomes cumbersome. For eg, for the c3d format has 5 part-downloads, so the folders will be as follows :
```
['~/.cache/huggingface/datasets/downloads/extracted/0e6bf028f490bf18c23ce572d1437c4ef32a74f630e33c26a806250d35cfcdd1', '~/.cache/huggingface/datasets/downloads/extracted/1b44fc5c7a6e031c904545422d449fd964f8ee795b9d1dcb0b6a76d03b50ebe6', '~/.cache/huggingface/datasets/downloads/extracted/137595188e96187c24ce1aa5c78200c7f78816fbd9d6c62354c01b3e6ec550c7', '~/.cache/huggingface/datasets/downloads/extracted/6c0c893e435f36fd79aa0f199f58fe16f01985f039644a7cb094a8c43a15ffd4', '~/.cache/huggingface/datasets/downloads/extracted/45e4703354cbc975e6add66f1b17b716c882b56f44575b033c5926aa5fcfb17f']
```
Each of these folders have a given set of subjects, so we'll be need to write extra code to fetch data from each of these folders, and the mpg format has 12 part-downloads which will lead to 12 folders having certain set of subjects, so it is cumbersome to process them.

I have added all the changes that were suggested. We just need to handle the multi-part download for c3d and mpg formats. Easiest way would be to have just one zip for these formats.

But we can handle this with a simple mapping that stores the id ranges (for each config), no? And an actual file path is not important during processing.

I have added code to handle c3d, mpg formats as well.  The data for the mpg format seems incomplete as it contains only 53 rows. I have added a note regarding this in the Data Splits section.

The real data test works fine and dummy_data test work fine. There were few missing files which was causing issues, I have fixed it now.

- Reduced the dummy_data size.
- Added sample dataset preprocessing code, it is not complete though.
- Added all changes suggested.

Let me know if anything else is required. Thank you. :)

Resolves #3457

Dataset Request : Add CMU Graphics Lab Motion Capture dataset [#3457](https://github.com/huggingface/datasets/issues/3457)
This PR adds the CMU MoCap Dataset.

The authors didn't respond even after multiple follow ups, so I ended up crawling the website to get categories, subcategories and description information. Some of the subjects do not have category/subcategory/description as well. I am using a subject to categories, subcategories and description map (metadata file).

Currently the loading of the dataset works for "asf/amc" and "avi" formats since they have a single download link. But "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Any suggestions/inputs on this would be helpful. Thank you.


[Minor edit] Fix typo in class name

Typo: `datasets.DatsetDict` -> `datasets.DatasetDict`


Add Nerval Metric

This PR adds readme.md and ner_val.py to metrics.
Nerval is a python package that helps evaluate NER models. It creates classification report and confusion matrix at entity level.


Fix `convert_file_size_to_int` for kilobits and megabits

_The documentation is not available anymore as the PR was closed or merged._

Minor change to fully align this function with the recent change in Transformers (https://github.com/huggingface/transformers/pull/16891)


Add Recall Metric Card

What this PR mainly does:
- add metric card for recall metric
- update docs in recall python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)


Add Precision Metric Card

What this PR mainly does:
- add metric card for precision metric
- update docs in precision python file

Note: I've also included a .json file with all of the metric card information. I've started compiling the relevant information in this type of .json files, and then using a script I wrote to generate the formatted metric card, as well as the docs to go in the .py file. I figured I'd upload the .json because it could be useful, especially if I also make a PR with the script I'm using (let me know if that's something you think would be beneficial!)


Fix some type annotation in doc


Update GH template for dataset viewer issues

You can see rendering at: https://github.com/huggingface/datasets/blob/6b48fedbdafe12a42c7b6edcecc32820af1a4822/.github/ISSUE_TEMPLATE/dataset-viewer.yml

Update template to use new issue forms instead.

With this PR we can check if this new feature is useful for us.

Once validated, we can update the other templates.

CC: @severo


Add to docs how to load from local script

This option was missing from the docs guide (it was only explained in the docstring of `load_dataset`). Although this is an infrequent use case, there might be some users interested in it.

Related to #4192

CC: @stevhliu


Cache miss during reload for datasets using image fetch utilities through map

Something isn't working

Hi ! Maybe one of the objects in the function is not deterministic across sessions ? You can read more about it and how to investigate here: https://huggingface.co/docs/datasets/about_cache

Hi @apsdehal! Can you verify that replacing
```python
def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": get_datasets_user_agent()},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
with
```python
USER_AGENT = get_datasets_user_agent()

def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": USER_AGENT},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image
```
fixes the issue?

Thanks @mariosasko. That does fix the issue. In general, I think these image downloading utilities since they are being used by a lot of image dataset should be provided as a part of `datasets` library right to keep the logic consistent and READMEs smaller? If they already exists, that is also great, please point me to those. I saw that `http_get` does exist.

You can find my rationale (and a proposed solution) for why these utilities are not a part of `datasets` here: https://github.com/huggingface/datasets/pull/4100#issuecomment-1097994003.

Makes sense. But, I think as the number of image datasets as grow, more people are copying pasting original code from docs to work as it is while we make fixes to them later. I think we do need a central place for these to avoid that confusion as well as more easier access to image datasets. Should we restart that discussion, possible on slack?

## Describe the bug

It looks like that result of `.map` operation dataset are missing the cache when you reload the script and always run from scratch. In same interpretor session, they are able to find the cache and reload it. But, when you exit the interpretor and reload it, the downloading starts from scratch.

## Steps to reproduce the bug

Using the example provided in `red_caps` dataset.
```python
from concurrent.futures import ThreadPoolExecutor
from functools import partial
import io
import urllib

import PIL.Image

import datasets
from datasets import load_dataset
from datasets.utils.file_utils import get_datasets_user_agent

def fetch_single_image(image_url, timeout=None, retries=0):
    for _ in range(retries + 1):
        try:
            request = urllib.request.Request(
                image_url,
                data=None,
                headers={"user-agent": get_datasets_user_agent()},
            )
            with urllib.request.urlopen(request, timeout=timeout) as req:
                image = PIL.Image.open(io.BytesIO(req.read()))
            break
        except Exception:
            image = None
    return image

def fetch_images(batch, num_threads, timeout=None, retries=0):
    fetch_single_image_with_args = partial(fetch_single_image, timeout=timeout, retries=retries)
    with ThreadPoolExecutor(max_workers=num_threads) as executor:
        batch["image"] = list(executor.map(lambda image_urls: [fetch_single_image_with_args(image_url) for image_url in image_urls], batch["image_url"]))
    return batch

def process_image_urls(batch):
    processed_batch_image_urls = []
    for image_url in batch["image_url"]:
        processed_example_image_urls = []
        image_url_splits = re.findall(r"http\S+", image_url)
        for image_url_split in image_url_splits:
            if "imgur" in image_url_split and "," in image_url_split:
                for image_url_part in image_url_split.split(","):
                    if not image_url_part:
                        continue
                    image_url_part = image_url_part.strip()
                    root, ext = os.path.splitext(image_url_part)
                    if not root.startswith("http"):
                      root = "http://i.imgur.com/" + root
                    root = root.split("#")[0]
                    if not ext:
                      ext = ".jpg"
                    ext = re.split(r"[?%]", ext)[0]
                    image_url_part = root + ext
                    processed_example_image_urls.append(image_url_part)
            else:
                processed_example_image_urls.append(image_url_split)
        processed_batch_image_urls.append(processed_example_image_urls)
    batch["image_url"] = processed_batch_image_urls
    return batch

dset = load_dataset("red_caps", "jellyfish")
dset = dset.map(process_image_urls, batched=True, num_proc=4)
features = dset["train"].features.copy()
features["image"] = datasets.Sequence(datasets.Image())
num_threads = 5
dset = dset.map(fetch_images, batched=True, batch_size=50, features=features, fn_kwargs={"num_threads": num_threads})
```

Run this in an interpretor or as a script twice and see that the cache is missed the second time.

## Expected results
At reload there should not be any cache miss

## Actual results
Every time script is run, cache is missed and dataset is built from scratch.

## Environment info
- `datasets` version: 2.1.1.dev0
- Platform: Linux-4.19.0-20-cloud-amd64-x86_64-with-glibc2.10
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.1


There is no dataset

## Dataset viewer issue for '*name of the dataset*'

**Link:** *link to the dataset viewer page*

*short description of the issue*

Am I the one who added this dataset ? Yes-No


Add remove_columns=True

Any reason why we can't just do `[inputs.copy()]` in this line for in-place operations to not have effects anymore:
https://github.com/huggingface/datasets/blob/bf432011ff9155a5bc16c03956bc63e514baf80d/src/datasets/arrow_dataset.py#L2232.

(in the `batched` case, we can also copy the inputs' values (list objects) to ignore in-place modifications to the inputs' columns)

I think `remove_columns=True` has no meaning, so I'm not a fan of this change.

@mariosasko copy does have a cost associated with it ... and plus you'll have to consider `deepcopy` Imagine columnds that are list of list of list of list .... Though I have to agree that `remove_columns=True` doesn't make sense (but, IMO, neither does it in its current use-case as it should refer to `input_columns`)

Okay closing this PR for the following reasons:
 - `remove_columns=True` was expected to keep the `.update`-like operator for `.map`. I initially thought it would be a good way to ignore function side effects and only keep output of that function (cf. PR description).
 - expected `remove_columns=True` is a bad API according to @mariosasko and introduces unecessary changes for little gain (strictly equivalent to `remove_columns=dset.column_names`)

This should fix all the issue we have with in place operations in mapping functions. This is crucial as where we do some weird things like:
```
def apply(batch):
    batch_size = len(batch["id"])
    batch["text"] = ["potato" for _ range(batch_size)]
    return {}

# Columns are: {"id": int}
dset.map(apply, batched=True, remove_columns="text") # crashes because `text` is not in the original columns
dset.map(apply, batched=True) # mapped datasets has `text` column
```

In this PR we suggest to have `remove_columns=True` so that we ignore the input completely, and just use the output to generate mapped dataset. This means that inplace operations won't have any effects anymore.


Embed image and audio files in `save_to_disk`

Following https://github.com/huggingface/datasets/pull/4184, currently a dataset saved using `save_to_disk` doesn't actually contain the bytes of the image or audio files. Instead it stores the path to your local files.

Adding `embed_external_files` and set it to True by default to save_to_disk would be kind of a breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:

- the resulting dataset is self contained, in case you want to delete your cache for example or share it with someone else
- users also upload these Arrow files to cloud storage via the fs parameter, and in this case they would expect to upload a self-contained dataset
- consistency with push_to_hub

This can be implemented at the same time as sharding for `save_to_disk` for efficiency, and reuse the helpers from `push_to_hub` to embed the external files.

cc @mariosasko


Support lists of multi-dimensional numpy arrays

Fix #4191.

CC: @SaulLu


Document save_to_disk and push_to_hub on images and audio files

Following https://github.com/huggingface/datasets/pull/4187, I explained in the documentation of `save_to_disk` and `push_to_hub` how they handle image and audio data.


load_dataset can't load local dataset,Unable to find ...

Something isn't working

Hi! :)

I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Hi @ahf876828330,

As @stevhliu pointed out, the proper way to load a dataset is not trying to load its metadata file.

In your case, as the dataset script is local, you should better point to your local loading script:
```python
dataset = load_dataset("dataset/opus_books.py")
```

Please, feel free to re-open this issue if the previous code snippet does not work for you.

> Hi! :)
>
> I believe that should work unless `dataset_infos.json` isn't actually a dataset. For Hugging Face datasets, there is usually a file named `dataset_infos.json` which contains metadata about the dataset (eg. the dataset citation, license, description, etc). Can you double-check that `dataset_infos.json` isn't just metadata please?

Yes，you are right!So if I have a metadata dataset local,How can I turn it to a dataset that can be used by the load_dataset() function？Are there some examples?

The metadata file isn't a dataset so you can't turn it into one. You should try @albertvillanova's code snippet above (now merged in the docs [here](https://huggingface.co/docs/datasets/master/en/loading#local-loading-script)), which uses your local loading script `opus_books.py` to:

1. Download the actual dataset.
2. Once the dataset is downloaded, `load_dataset` will load it for you.

Traceback (most recent call last):
  File "/home/gs603/ahf/pretrained/model.py", line 48, in <module>
    dataset = load_dataset("json",data_files="dataset/dataset_infos.json")
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1675, in load_dataset
    **config_kwargs,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1496, in load_dataset_builder
    data_files=data_files,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 1155, in dataset_module_factory
    download_mode=download_mode,
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/load.py", line 800, in get_module
    data_files = DataFilesDict.from_local_or_remote(patterns, use_auth_token=self.downnload_config.use_auth_token)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 582, in from_local_or_remote
    if not isinstance(patterns_for_key, DataFilesList)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 544, in from_local_or_remote
    data_files = resolve_patterns_locally_or_by_urls(base_path, patterns, allowed_extensions)
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 194, in resolve_patterns_locally_or_by_urls
    for path in _resolve_single_pattern_locally(base_path, pattern, allowed_extensions):
  File "/home/gs603/miniconda3/envs/coderepair/lib/python3.7/site-packages/datasets/data_files.py", line 144, in _resolve_single_pattern_locally
    raise FileNotFoundError(error_msg)
FileNotFoundError: Unable to find '/home/gs603/ahf/pretrained/dataset/dataset_infos.json' at /home/gs603/ahf/pretrained

![image](https://user-images.githubusercontent.com/33253979/164413285-84ea65ac-9126-408f-9cd2-ce4751a5dd73.png)
![image](https://user-images.githubusercontent.com/33253979/164413338-4735142f-408b-41d9-ab87-8484de2be54f.png)

the code is in the model.py,why I can't use the load_dataset function to load my local dataset?


feat: create an `Array3D` column from a list of arrays of dimension 2

enhancement

New feature or request

Hi @SaulLu, thanks for your proposal.

Just I got a bit confused about the dimensions...
- For the 2D case, you mention it is possible to create an `Array2D` from a list of arrays of dimension 1
- However, you give an example of creating an `Array2D` from arrays of dimension 2:
  - the values of `data_map` are arrays of dimension 2
  - the outer list in `prepare_dataset_2D` should not be taken into account in the dimension counting, as it is used because in `map` you pass `batched=True`

Note that for the 3D alternatives you mention:
- In `prepare_dataset_3D_ter`, you create an `Array3D` from arrays of dimension 3:
  - the array `data_map[index][np.newaxis, :, :]` has dimension 3
  - the outer list in `prepare_dataset_3D_ter` is the one used by `batched=True`
- In `prepare_dataset_3D_bis`, you create an `Array3D` from a list of list of lists:
  - the value of `data_map[index].tolist()` is a list of lists
  - it is enclosed by another list `[data_map[index].tolist()]`, thus giving a list of list of lists
  - the outer list is the one used by `batched=True`

Therefore, if I understand correctly, your request would be to be able to create an `Array3D` from a list of an array of dimension 2:
- In `prepare_dataset_3D`, `data_map[index]` is an array of dimension 2
- it is enclosed by a list `[data_map[index]]`, thus giving a list of an array of dimension 2
- the outer list is the one used by `batched=True`

Please, feel free to tell me if I did not understand you correctly.

Hi @albertvillanova ,

Indeed my message was confusing and you guessed right :smile: : I think would be interesting to be able to create an Array3D from a list of an array of dimension 2.

For the 2D case I should have given as a "similar" example:
```python

data_map_1D = {
    1: np.array([0.2, 0.4]),
    2: np.array([0.1, 0.4]),
}

def prepare_dataset_2D(batch):
    batch["pixel_values"] = [[data_map_1D[index]] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(1, 2), dtype="float32")})
)
```

**Is your feature request related to a problem? Please describe.**

It is possible to create an `Array2D` column from a list of arrays of dimension 1. Similarly, I think it might be nice to be able to create a `Array3D` column from a list of lists of arrays of dimension 1.

To illustrate my proposal, let's take the following toy dataset t:
```python
import numpy as np
from datasets import Dataset, features

data_map = {
    1: np.array([[0.2, 0,4],[0.19, 0,3]]),
    2: np.array([[0.1, 0,4],[0.19, 0,3]]),
}

def create_toy_ds():
    my_dict = {"id":[1, 2]}
    return Dataset.from_dict(my_dict)

ds = create_toy_ds()
```

The following 2D processing works without any errors raised:
```python
def prepare_dataset_2D(batch):
    batch["pixel_values"] = [data_map[index] for index in batch["id"]]
    return batch

ds_2D = ds.map(
    prepare_dataset_2D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array2D(shape=(2, 3), dtype="float32")})
)
```

The following 3D processing doesn't work:
```python
def prepare_dataset_3D(batch):
    batch["pixel_values"] = [[data_map[index]] for index in batch["id"]]
    return batch

ds_3D = ds.map(
    prepare_dataset_3D,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3, dtype="float32")})
)
```
The error raised is:
```
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
[<ipython-input-6-676547e4cd41>](https://localhost:8080/#) in <module>()
      3     batched=True,
      4     remove_columns=ds.column_names,
----> 5     features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
      6 )

12 frames
[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   1971                 new_fingerprint=new_fingerprint,
   1972                 disable_tqdm=disable_tqdm,
-> 1973                 desc=desc,
   1974             )
   1975         else:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    518             self: "Dataset" = kwargs.pop("self")
    519         # apply actual function
--> 520         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    521         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    522         for dataset in datasets:

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    485         }
    486         # apply actual function
--> 487         out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    488         datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    489         # re-apply format to the output

[/usr/local/lib/python3.7/dist-packages/datasets/fingerprint.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
    456             # Call actual function
    457
--> 458             out = func(self, *args, **kwargs)
    459
    460             # Update fingerprint of in-place transforms + update in-place history of transforms

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in _map_single(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset, disable_tqdm, desc, cache_only)
   2354                                 writer.write_table(batch)
   2355                             else:
-> 2356                                 writer.write_batch(batch)
   2357                 if update_data and writer is not None:
   2358                     writer.finalize()  # close_stream=bool(buf_writer is None))  # We only close if we are writing in a file

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in write_batch(self, batch_examples, writer_batch_size)
    505             col_try_type = try_features[col] if try_features is not None and col in try_features else None
    506             typed_sequence = OptimizedTypedSequence(batch_examples[col], type=col_type, try_type=col_try_type, col=col)
--> 507             arrays.append(pa.array(typed_sequence))
    508             inferred_features[col] = typed_sequence.get_inferred_type()
    509         schema = inferred_features.arrow_schema if self.pa_writer is None else self.schema

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._handle_arrow_array_protocol()

[/usr/local/lib/python3.7/dist-packages/datasets/arrow_writer.py](https://localhost:8080/#) in __arrow_array__(self, type)
    175                     storage = list_of_np_array_to_pyarrow_listarray(data, type=pa_type.value_type)
    176                 else:
--> 177                     storage = pa.array(data, pa_type.storage_dtype)
    178                 return pa.ExtensionArray.from_storage(pa_type, storage)
    179

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib.array()

/usr/local/lib/python3.7/dist-packages/pyarrow/array.pxi in pyarrow.lib._sequence_to_array()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

/usr/local/lib/python3.7/dist-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowInvalid: Can only convert 1-dimensional array values
```

**Describe the solution you'd like**
No error in the second scenario and an identical result to the following snippets.

**Describe alternatives you've considered**
There are other alternatives that work such as:
```python

def prepare_dataset_3D_bis(batch):
    batch["pixel_values"] = [[data_map[index].tolist()] for index in batch["id"]]
    return batch

ds_3D_bis = ds.map(
    prepare_dataset_3D_bis,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1, 2, 3), dtype="float32")})
)
```
or
```python
def prepare_dataset_3D_ter(batch):
    batch["pixel_values"] = [data_map[index][np.newaxis, :, :] for index in batch["id"]]
    return batch

ds_3D_ter = ds.map(
    prepare_dataset_3D_ter,
    batched=True,
    remove_columns=ds.column_names,
    features=features.Features({"pixel_values": features.Array3D(shape=(1,  2, 3), dtype="float32")})
)
```
But both solutions require the user to be aware that `data_map[index]` is an `np.array` type.

cc @lhoestq as we discuss this offline :smile:


Deprecate `shard_size` in `push_to_hub` in favor of `max_shard_size`

This PR adds a `max_shard_size` param to `push_to_hub` and deprecates `shard_size` in favor of this new param to have a more descriptive name (a shard has at most the `shard_size` bytes in `push_to_hub`) for the param and to align the API with [Transformers](https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/src/transformers/modeling_utils.py#L1350).


Document how to use FAISS index for special operations, by accessing the index itself.

Close #4029.


Support streaming cnn_dailymail dataset

Did you run the `datasets-cli` command before merging to make sure you generate all the examples ?

Support streaming cnn_dailymail dataset.

Fix #3969.

CC: @severo


Don't duplicate data when encoding audio or image

_The documentation is not available anymore as the PR was closed or merged._

I'm not familiar with the concept of streaming vs non-streaming in HF datasets. I just wonder that you have the distinction here. Why doesn't it work to always make use of `bytes`? "using a local file - which is often required for audio" - why would that be?

The `path` would always point to some location in the `cache_dir`? I think this can be problematic. I would have expected that after I did `dataset.save_to_disk(...)` that I can remove the cache dir. But maybe just because I'm not familiar with HF. Or maybe the docs can be improved to clarify this.

We could always load every data file into `bytes` and save it this way the audio as bytes in `arrow` format, but the problem then would be that it makes the `file` column useless, *i.e.* people cannot inspect the audio file locally anymore or else they would need to first save bytes as a file which is not evident. This either breaks backwards compatibility or forces the user to stored 2x the required size locally. There was a longer discussion here: https://github.com/huggingface/datasets/issues/3663

It's a good argument though that `dataset.save_to_disk(...)` should save everything that is needed to the disk and should be independent of other folders, but I do think the arguments of #3663 to not break backwards compatibility and to allow people to inspect the downloaded audio files locally are a bit more important here.

But maybe, we could add a flag, `save_files_as_bytes` or `make_independent`, `make_self_contained` or a better name to `save_to_disk(...)` and `push_to_hub(...)` that would allow to make the resulting folder completely independent.

What do you think @mariosasko @lhoestq @polinaeterna @anton-l ?

For context: you can either store the path to local images or audio files, or the bytes of those files.

If your images and audio files are local files, then the arrow file from `save_to_disk` will store paths to these files.
If you want to include the bytes or your images or audio files instead, you must `read()` those files first.
This can be done by storing the "bytes" instead of the "path" of the images or audio files.

On the other hand, the resulting Parquet files from `push_to_hub` are self-contained, so that anyone can reload the dataset from the Hub. If your dataset contains image or audio data, the Parquet files will store the bytes of your images or audio files.

For now I just updated the documentation: https://github.com/huggingface/datasets/pull/4193. Maybe we can also embed the image and audio bytes in `save_to_disk` when we implement sharding, so that is can be done as efficiently as `push_to_hub`.

Anyway, merging this one :)

Right now if you pass both the `bytes` and a local `path` for audio or image data, then the `bytes` are unnecessarily written in the Arrow file, while we could just keep the local `path`.

This PR discards the `bytes` when the audio or image file exists locally.

In particular it's common for audio datasets builders to provide both the bytes and the local path in order to work for both streaming (using the bytes) and non-streaming mode (using a local file - which is often required for audio).

cc @patrickvonplaten


Fix outdated docstring about default dataset config


Librispeech documentation, clarification on format

The documentation in the code is definitely outdated - thanks for letting me know, I'll remove it in https://github.com/huggingface/datasets/pull/4184 .

You're exactly right `audio` `array` already decodes the audio file to the correct waveform. This is done on the fly, which is also why one should **not** do `ds["audio"]["array"][0]` as this will decode all dataset samples, but instead `ds[0]["audio"]["array"]` see: https://huggingface.co/docs/datasets/audio_process#audio-datasets

So, again to clarify: On disk, only the raw flac file content is stored? Is this also the case after `save_to_disk`?

And is it simple to also store it re-encoded as ogg or mp3 instead?

Hey,

Sorry yeah I was just about to look into this! We actually had an outdated version of Librispeech ASR that didn't save any files, but instead converted the audio files to a byte string, then was then decoded on-the-fly. This however is not very user-friendly so we recently decided to instead show the full path of the audio files with the `path` parameter.

I'm currently changing this for Librispeech here: https://github.com/huggingface/datasets/pull/4184 .
You should be able to see the audio file in the original `flac` format under `path` then. I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?

> I don't think it's a good idea to convert to MP3 out-of-the-box, but we could maybe think about some kind of convert function for audio datasets cc @lhoestq ?

Sure, I would expect that `load_dataset("librispeech_asr")` would give you the original (not re-encoded) data (flac or already decoded). So such re-encoding logic would be some separate generic function. So I could do sth like `dataset.reencode_as_ogg(**ogg_encode_opts).save_to_disk(...)` or so.

A follow-up question: I wonder whether a Parquet dataset is maybe more what we actually want to have? (Following also my comment here: https://github.com/huggingface/datasets/pull/4184#issuecomment-1105045491.) Because I think we actually would prefer to embed the data content in the dataset.

So, instead of `save_to_disk`/`load_from_disk`, we would use `to_parquet`,`from_parquet`? Is there any downside? Are arrow files more efficient?

Related is also the doc update in #4193.

`save_to_disk` saves the dataset as an Arrow file, which is the format we use to load a dataset using memory mapping. This way the dataset does not fill your RAM, but is read from your disk instead.

Therefore you can directly reload a dataset saved with `save_to_disk` using `load_from_disk`.

Parquet files are used for cold storage: to use memory mapping on a Parquet dataset, you first have to convert it to Arrow. We use Parquet to reduce the I/O when pushing/downloading data from the Hugging face Hub. When you load a Parquet file from the Hub, it is converted to Arrow on the fly during the download.

https://github.com/huggingface/datasets/blob/cd3ce34ab1604118351e1978d26402de57188901/datasets/librispeech_asr/librispeech_asr.py#L53

> Note that in order to limit the required storage for preparing this dataset, the audio
> is stored in the .flac format and is not converted to a float32 array. To convert, the audio
> file to a float32 array, please make use of the `.map()` function as follows:
>
> ```python
> import soundfile as sf
> def map_to_array(batch):
>     speech_array, _ = sf.read(batch["file"])
>     batch["speech"] = speech_array
>     return batch
> dataset = dataset.map(map_to_array, remove_columns=["file"])
> ```

Is this still true?

In my case, `ds["train.100"]` returns:
```
Dataset({
    features: ['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'],
    num_rows: 28539
})
```
and taking the first instance yields:
```
{'file': '374-180298-0000.flac',
 'audio': {'path': '374-180298-0000.flac',
  'array': array([ 7.01904297e-04,  7.32421875e-04,  7.32421875e-04, ...,
         -2.74658203e-04, -1.83105469e-04, -3.05175781e-05]),
  'sampling_rate': 16000},
 'text': 'CHAPTER SIXTEEN I MIGHT HAVE TOLD YOU OF THE BEGINNING OF THIS LIAISON IN A FEW LINES BUT I WANTED YOU TO SEE EVERY STEP BY WHICH WE CAME I TO AGREE TO WHATEVER MARGUERITE WISHED',
 'speaker_id': 374,
 'chapter_id': 180298,
 'id': '374-180298-0000'}
```

The `audio` `array` seems to be already decoded. So such convert/decode code as mentioned in the doc is wrong?

But I wonder, is it actually stored as flac on disk, and the decoding is done on-the-fly? Or was it decoded already during the preparation and is stored as raw samples on disk?

Note that I also used `datasets.load_dataset("librispeech_asr", "clean").save_to_disk(...)` and then `datasets.load_from_disk(...)` in this example. Does this change anything on how it is stored on disk?

A small related question: Actually I would prefer to even store it as mp3 or ogg on disk. Is this easy to convert?


[Librispeech] Add 'all' config

Just that I understand: With this change, simply doing `load_dataset("librispeech_asr")` is possible and returns the whole dataset?

And to get the subsets, I do sth like:
```python
ds = load_dataset("librispeech_asr")
train_ds = ds["train"]
dev_clean_ds = ds["dev-clean"]
dev_other_ds = ds["dev-other"]
test_clean_ds = ds["test-clean"]
test_other_ds = ds["test-other"]
```
?

> Just that I understand: With this change, simply doing `load_dataset("librispeech_asr")` is possible and returns the whole dataset?
>
> And to get the subsets, I do sth like:
>
> ```python
> ds = load_dataset("librispeech_asr")
> train_ds = ds["train"]
> dev_clean_ds = ds["dev-clean"]
> dev_other_ds = ds["dev-other"]
> test_clean_ds = ds["test-clean"]
> test_other_ds = ds["test-other"]
> ```
>
> ?

You could do:

```python
ds = load_dataset("librispeech_asr", "all")  # <- note that we have to pass a config
train_ds = ds["train"]
dev_clean_ds = ds["dev-clean"]
dev_other_ds = ds["dev-other"]
test_clean_ds = ds["test-clean"]
test_other_ds = ds["test-other"]
```

So, `load_dataset("librispeech_asr")` is not possible, it must be `load_dataset("librispeech_asr", "all")`?

Why is that?

The docs say:
```
name: `str` name, optional configuration for the dataset that affects the data generated on disk. Different
    `builder_config`s will have their own subdirectories and versions.
    If not provided, uses the first configuration in self.BUILDER_CONFIGS
```
https://github.com/huggingface/datasets/blob/cd3ce34ab1604118351e1978d26402de57188901/src/datasets/builder.py#L228

Or maybe you could just define `DEFAULT_CONFIG_NAME`?

> If not provided, uses the first configuration in self.BUILDER_CONFIGS

Oh crap this is outdated documentation. No it doesn't take the first config by default.

EDIT: opened a PR to fix this: https://github.com/huggingface/datasets/pull/4186

> No it doesn't take the first config by default.

But defining `DEFAULT_CONFIG_NAME` would work?

So should we define `DEFAULT_CONFIG_NAME = "all"` here as well? I think this is a reasonable default config.

Don't most datasets have some default config?

> But defining DEFAULT_CONFIG_NAME would work?
>
> So should we define DEFAULT_CONFIG_NAME = "all" here as well? I think this is a reasonable default config.

Yes that would work, and I also find it reasonable to do it :)

> Don't most datasets have some default config?

Most datasets only have one configuration, so the single configuration is the default one. Then other datasets gave several configurations, and whether they have a default one is decided case-by-case.

e.g. `glue` is a benchmark and doesn't have a default task, one must choose which task of `glue` they want to use explicitely.

Thanks a lot for the feedback!

Using `"all"` now as the default config. I changed the layout a bit so that there is not a single "train", but instead we have multiple "train.clean.100", "train.clean.360", "train.other.500". This way we don't even need to do filtering and it's also cleaner IMO.

@albertz - you should now be able to do the following:

```python
load_dataset("librispeech_asr")  # <- run this once to download, prepare dataset and cache everything

# The following operations will be very fast since all the downloading and processing is already cached
train_1 = load_dataset("librispeech_asr", split="train.clean.100")
print(train_1)
train_2 = load_dataset("librispeech_asr", split="train.clean.100+train.clean.360")
print(train_2)
train_full = load_dataset("librispeech_asr", split="train.clean.100+train.clean.360+train.other.500")
print(train_full)
dev_clean_ds = load_dataset("librispeech_asr", split="validation.clean")
print(dev_clean_ds)
dev_other_ds = load_dataset("librispeech_asr", split="validation.other")
print(dev_other_ds)
test_clean_ds = load_dataset("librispeech_asr", split="test.clean")
print(test_clean_ds)
test_other_ds = load_dataset("librispeech_asr", split="test.other")
print(test_other_ds)
```

Think this way we have the best of both worlds. Also @lhoestq, I think we could highlight better in the docs that it's possible to combine different splits. We do this actually quite a lot for speech. For Common Voice many people include "validation" in the training if the data is too small, e.g.: https://github.com/huggingface/transformers/blob/ff06b177917384137af2d9585697d2d76c40cdfc/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py#L147

Should we maybe add a short section to the loading tutorial here: https://huggingface.co/docs/datasets/v2.1.0/en/loading#hugging-face-hub ? (Happy to do it)

Is there any advantage or difference in calling `load_dataset` multiple times for each split? Or why not just call `load_dataset` once and then access each split?

Note in our case, we cannot really use the caching mechanism because we have a recipe pipeline used by multiple users (and I think a common cache dir for all users might end up in problems) and we basically would use `load_dataset("librispeech_asr").save_to_disk(...)` and then later `load_from_disk(...)`. (See here: https://github.com/rwth-i6/i6_core/pull/253)

So with `load_from_disk`, we cannot really provide the split this way, so we anyway would do sth like:
```python
ds = datasets.load_from_disk(...)
train = ds["train"]
```
Or with your latest proposal, it would look like:
```python
ds = datasets.load_from_disk(...)
train_ds = datasets.concatenate_datasets(
  [ds["train.clean.100"], ds["train.clean.360"], ds["train.other.500"]])
```
right?

> Is there any advantage or difference in calling `load_dataset` multiple times for each split? Or why not just call `load_dataset` once and then access each split?
>
> Note in our case, we cannot really use the caching mechanism because we have a recipe pipeline used by multiple users (and I think a common cache dir for all users might end up in problems) and we basically would use `load_dataset("librispeech_asr").save_to_disk(...)` and then later `load_from_disk(...)`. (See here: [rwth-i6/i6_core#253](https://github.com/rwth-i6/i6_core/pull/253))
>
> So with `load_from_disk`, we cannot really provide the split this way, so we anyway would do sth like:
>
> ```python
> ds = datasets.load_from_disk(...)
> train = ds["train"]
> ```
>
> Or with your latest proposal, it would look like:
>
> ```python
> ds = datasets.load_from_disk(...)
> train_ds = datasets.concatenate_datasets(
>   [ds["train.clean.100"], ds["train.clean.360"], ds["train.other.500"]])
> ```
>
> right?

I see the use case! The only advantage by calling `datasets` multiple times is that one can easily "merge" splits with `"+"`, but yeah you can do the exact same with `concatenate`.

@lhoestq what do you think is the best approach with `load_from_disk`?

@albertz, you could also define the `cache_dir` when doing `load_dataset(...)` which will then put all the relevant `arrow` files int the cache dir that you defined, e.g.:

```python
load_dataset("librispeech_asr", cache_dir="/easy/to/access/directory")
```

@albertz, I took a read through https://github.com/rwth-i6/i6_core/pull/253 .

I think the best would be the following:

1. Do `ds = load_dataset(..., cache_dir="/dir/that/is/easy/to/access")` <- having merged this PR, this will save all the original `.flac` files in the `cache_dir`
2. Do `ds.save_to_disk("local/path")` this should then only save the `arrow.format` with a `path` string to the audio files which are located in `cache_dir` <- this won't require a lot of memory after https://github.com/huggingface/datasets/pull/4184#discussion_r854132740 is fixed and can be done for each person individually.
3. `ds = datasets.load_from_disk("local/path")` can the be used. An object of `ds` will then have a `path` variable that links to the original audio files in the `cache_dir`. You can change these audio files then easily to `.mp3. You could do this with the `.map(...)` function, e.g. define a function that maps through all audio files, load them and then save them on disk afterward.

@lhoestq - I think this one is good to go

> @albertz, I took a read through [rwth-i6/i6_core#253](https://github.com/rwth-i6/i6_core/pull/253) .
>
> I think the best would be the following:
>
> 1. Do `ds = load_dataset(..., cache_dir="/dir/that/is/easy/to/access")` <- having merged this PR, this will save all the original `.flac` files in the `cache_dir`
> 2. Do `ds.save_to_disk("local/path")` this should then only save the `arrow.format` with a `path` string to the audio files which are located in `cache_dir` <- this won't require a lot of memory after [[Librispeech] Add 'all' config #4184 (comment)](https://github.com/huggingface/datasets/pull/4184#discussion_r854132740) is fixed and can be done for each person individually.
> 3. `ds = datasets.load_from_disk("local/path")` can the be used. An object of `ds` will then have a `path` variable that links to the original audio files in the `cache_dir`. You can change these audio files then easily to `.mp3. You could do this with the `.map(...)` function, e.g. define a function that maps through all audio files, load them and then save them on disk afterward.

Oh, so you say that our current implementation in https://github.com/rwth-i6/i6_core/pull/253 is broken? Because our cache dir is just some temp directory which will be removed afterwards, and we just store what we get out of `save_to_disk`. I think it would be good to clarify that in the doc of `save_to_disk`, that this is not enough and can depend on files from the cache dir. (@dthulke)

So, you say we anyway need to share the cache dir among users? But we would want to make sure that after the initial download and preparation of the data, this is set to readonly, because we want to make sure that other people will not modify the data in any way. Right?

But then, we don't really need the `save_to_disk` and `load_from_disk` at all, right?

@albertz

> Oh, so you say that our current implementation in https://github.com/rwth-i6/i6_core/pull/253 is broken? Because our cache dir is just some temp directory which will be removed afterwards, and we just store what we get out of save_to_disk. I think it would be good to clarify that in the doc of save_to_disk, that this is not enough and can depend on files from the cache dir. (@dthulke)

Oh, I wasn't aware that audio files are handled this way. Then we should have the cache directory as an additional job output, so that we keep the audio files.

> So, you say we anyway need to share the cache dir among users?

No, the cache dir can still be a directory in the job output folder. Then the audio paths in the corresponding dataset column correspond to the flac files in that directory. This way the "output" of the job is contained into the job directory and we don't write files to a global cache directory that is independent of the sisyphus graph.

If we want to share the audio data between different users, we can just link to a central instance of the job (similar to how we do it with the `DownloadLibriSpeechCorpusJob`).

@dthulke - that's a good point actually! So you can do both things:

1. Convert all audio files to bytes. Bytes can be saved by `arrow` so in this case you can do `save_to_disk(...)`, but then you cannot really inspect the audio files locally as they'll just be saved within a large arrow file (this actually used to be the default case but we're changing this now). The problem of this is summarized here a bit:  https://github.com/huggingface/datasets/issues/3663 . You can still do this if you'd like, e.g. you could do:

```python
ds = load_dataset("librispeech_asr")

def read_file(batch):
    with open(batch["file"], "r") as f:
        batch["bytes"] = f.read()
    return batch

ds = ds.map(read_file)
ds.save_to_disk("/path") <- the saved arrow object will now contain everything you need
```

however this is not recommend - it's should be much easier to just save the path to the downloaded audio files.

2. Not convert audio files to bytes, but just leave them in their original file format. Then only the path to the original files will be save in arrow. This will be the default case. This means that when you do `load_dataset(...)` both the orginal audio data and the arrow file will be saved in the `cache_dir` (which can be saved locally for every user or in a shared cache - we actually use a shared cache quite a bit at Hugging Face). When do you do `save_to_disk(...)` now only the `path` will be saved in `arrow` format (after this PR is merged, you'll see that the `arrow files should be very light weight` meaning that `save_to_disk(...)` can be done for every user, but has a dependency on the `cache_dir` (because the audio files live there).

=> Now what you could do as well would be to simply move all the audio files to the folder you want (the `save_to_disk(...)` folder) and then change the path of every sample to this folder (maybe with `map(...)`) and then this folder would be self contained. I do however think it's better to just specific a `cache_dir` and re-use `load_dataset(...)` every time instead of `load_from_disk` or `save_to_disk(...)`. Note that you can even pass the relevant cache files to `load_dataset(...)` here: https://huggingface.co/docs/datasets/v2.1.0/en/package_reference/loading_methods#datasets.load_dataset.data_files in which case you can be 100% sure that nothing is redownloaded.

We discussed storing audio files quite a bit, e.g. see: https://github.com/huggingface/datasets/issues/3663 and had (too many) changes around this topic recently, but we've come to the conclusion that the best is to leave the audio format in the format it was originally (`.flac` for Librispeech) so that the user can easily inspect it / understand the data. Arrow cannot save data is `.flac` so we'll just save a path to the original data. Curious to hear your guys opinion on this as well.

So what I would suggest here is to do the following:

1. Do `load_dataset(..., cache_dir=/a/read-only/folder)`
2.
- Either just re-use `load_dataset(..., cache_dir=...)` which should always re-use the data in the `cache_dir` since the hash of the url matches - so there should never be any duplicated downloading

or

- If you want to store the files in MP3 locally, first convert the files to MP3 in the read-only folder, then take do `ds.save_to_disk(/some/path)` which will save the correct path to the read-only folder to MP3 and then you can easily re-use the small arrow dataset that is saved in `/some/path`

> So what I would suggest here is to do the following:
>
> 1. Do `load_dataset(..., cache_dir=/a/read-only/folder)`
>
> * Either just re-use `load_dataset(..., cache_dir=...)` which should always re-use the data in the `cache_dir` since the hash of the url matches - so there should never be any duplicated downloading
>
> or
>
> * If you want to store the files in MP3 locally, first convert the files to MP3 in the read-only folder, then take do `ds.save_to_disk(/some/path)` which will save the correct path to the read-only folder to MP3 and then you can easily re-use the small arrow dataset that is saved in `/some/path`

Also relevant here: https://github.com/huggingface/datasets/issues/3663

I also added some documentation about how `save_to_disk` handles audio files here: https://github.com/huggingface/datasets/pull/4193

> > So, you say we anyway need to share the cache dir among users?
>
> No, the cache dir can still be a directory in the job output folder.

@dthulke But this is what I mean. When we share the job output folder, it means we share the cache dir among users.

I wonder if `load_dataset(..., cache_dir=job_output_cache_dir)` is always save to do then, that it really would not modify the `job_output_cache_dir`.

We could enforce that by making the `job_output_cache_dir` read-only afterwards. We currently don't do this.

@patrickvonplaten @dthulke But in any case, we actually prefer the data content to be inside the dataset (the arrow files). Lots of small files would be very problematic for our cache manager. We have one main copy of the data on NFS, but accessing the NFS directly by all computing nodes is not feasible, so the cache manager will have copies of the files on the nodes. So it means, whenever we access some file, we query the cache manager DB whether the file is already cached somewhere (some other computing node) and if so, it copies it from the other computing node and not from NFS. This works very well when there are not too many files (but the files can be big). So, we want to have only a few but big files. Even for NFS access this is much better.

I also commented in #3663.

Hey @albertz @dthulke,

Thanks a lot for your input!

We've discussed quite a bit with @lhoestq and we think the best approach is the following:

a)
`load_dataset(...)` will not store both bytes and the files because this would mean that 3x the size of the dataset would often be needed (1. the compressed `tar.gz` file, 2. the extracted file b, 3. the raw bytes in arrow format).

For canonical datasets like librispeech and common voice I think we want to keep the dataset filenames because of i) no breaking changes and ii) reasons explained in #3663

However it's also trivial to write your own datasetset downloading script of librispeech and just not extract the folder e.g. this line: https://huggingface.co/datasets/common_voice/blob/main/common_voice.py#L671

And then it'll be allowed to save the bytes and the dataset will be self-contained out-of-the-box when using `load_dataset(...)`

b) Now, one major problem that you guys uncovered is that `save_to_disk(...)` is currently not necessarily saving a dataset to be self-contained. We will change that asap. This means that after we've corrected this when you do download the canonical librispeech dataset the following will work:

```python
ds = load_dataset("....")  # <- here we have a dependency on the filepathes
ds[0]["audio"]["bytes"] # <- will not work

ds.save_to_disk("/local/path")  # <- now we want to have a self-contained dataset in arrow format, so we load the files into bytes and save it in arrow format

# now you can delete everything besides "/local/path"

ds = load_from_disk("/local/path")  # <- this will work
```

So either option a) where you define your own librispeech data downloading script (you guys could just sign up here: https://huggingface.co/join) and upload a dataset loading script in private mode so that no one can see it and you would always store the audio as bytes or b) where you first load then save to disk then delete cache would work.

Hope that fits in your vision :-)

cc @lhoestq @mariosasko

@patrickvonplaten sounds like a good approach to me. For b) this could even be configurable with a parameter like `embed_external_files` as you have for `push_to_hub` (if people prefer to keep separate audio files).

> However it's also trivial to write your own datasetset downloading script of librispeech and just not extract the folder

I don't exactly understand. In all cases, we need to extract it to prepare the dataset, or not? No matter if we want to store the raw bytes inside the dataset or leaving them as local files. Just in the first case, we can safely delete the extracted files after the dataset preparation.

> `save_to_disk(...)` is currently not necessarily saving a dataset to be self-contained. We will change that asap.

For us, this sounds exactly like what we want.

But regarding not introducing breaking changes, wouldn't this maybe also break some setups for users who don't expect this new behavior?

@albertz I would suggest to move the discussion on implementation details on our side to the following issue: rwth-i6/i6_core/issues/257

I like the idea of adding `embed_external_files` and set it to True by default to `save_to_disk`.
It's indeed a kind of breaking change since some users will get bigger Arrow files when updating the lib, but the advantages are nice:
1. I like the idea of having it self contained, in case you want to delete your cache
2. users also upload these Arrow files to cloud storage via the `fs` parameter, and in this case they would expect to upload a self-contained dataset
3. consistency with `push_to_hub`

If it sounds good to you I'll open an issue to discuss this and track the advancements

Add `"all"` config to Librispeech


Document librispeech configs

I think the main purpose of #4179 was how to be able to load both configs into one, so should we maybe add this part of the code: https://github.com/huggingface/datasets/issues/4179#issuecomment-1102383717

to the doc?

Actually @lhoestq would this work given that they have different split names: https://huggingface.co/datasets/librispeech_asr#data-splits ?

This doc extension does not explain why I can't simply load the whole dataset. Or what workaround I need to get the whole dataset, which is what people usually want for Librispeech.

_The documentation is not available anymore as the PR was closed or merged._

@lhoestq, I can add a `"all"` config to Librispeech have the datasets already cached somewhere

I'm closing this PR then, feel free to continue the discussion in https://github.com/huggingface/datasets/issues/4179

Added an example of how to load one config or the other


Zenodo.org download is not responding

bug

Something isn't working

[Off topic but related: Is the uptime of S3 provably better than Zenodo's?]

Hi @dkajtoch, please note that at HuggingFace we are not hosting this dataset: we are just using a script to download their data file and create a dataset from it.

It was the dataset owners decision to host their data at Zenodo. You can see this on their website: https://marcobaroni.org/composes/sick.html

And yes, you are right: Zenodo is currently having some incidents and people are reporting problems from it.

On the other hand, we could contact the data owners and propose them to host their data at our Hugging Face Hub.

@julien-c I guess so.

Thanks @albertvillanova. I know that the problem lies in the source data. I just wanted to point out that these kind of problems are unavoidable without having one place where data sources are cached. Websites may go down or data sources may move. Having a copy in Hugging Face Hub would be a great solution.

Definitely, @dkajtoch! But we have to ask permission to the data owners. And many dataset licenses directly forbid data redistribution: in those cases we are not allowed to host their data on our Hub.

Ahhh good point! License is the problem :(

## Describe the bug
Source download_url from zenodo.org does not respond.
`_DOWNLOAD_URL = "https://zenodo.org/record/2787612/files/SICK.zip?download=1"`
Other datasets also use zenodo.org to store data and they cannot be downloaded as well.

It would be better to actually use more reliable way to store original data like s3 bucket.

## Steps to reproduce the bug
```python
load_dataset("sick")
```

## Expected results
Dataset should be downloaded.

## Actual results
ConnectionError: Couldn't reach https://zenodo.org/record/2787612/files/SICK.zip?download=1 (ReadTimeout(ReadTimeoutError("HTTPSConnectionPool(host='zenodo.org', port=443): Read timed out. (read timeout=100)")))

## Environment info
- `datasets` version: 2.1.0
- Platform: Darwin-21.4.0-x86_64-i386-64bit
- Python version: 3.7.11
- PyArrow version: 7.0.0
- Pandas version: 1.3.5


Support streaming FLEURS dataset

dataset bug

A bug in a dataset script provided in the library

Yes, you just have to use `dl_manager.iter_archive` instead of `dl_manager.download_and_extract`.

That's because `download_and_extract` doesn't support TAR archives in streaming mode.

Tried to make it streamable, but I don't think it's really possible. @lhoestq @polinaeterna maybe you guys can check:
https://huggingface.co/datasets/google/fleurs/commit/dcf80160cd77977490a8d32b370c027107f2407b

real quick.

I think the problem is that we cannot ensure that the metadata file is found before the audio. Or is this possible somehow @lhoestq ?

@patrickvonplaten I think the metadata file should be found first because the audio files are contained in a folder next to the metadata files (just as in common voice), so the metadata files should be "on top of the list"   as they are closer to the root in the directories hierarchy

@patrickvonplaten but apparently it doesn't... I don't really know why.

Yeah! Any ideas what could be the reason here? cc @lhoestq ?

The order of the files is determined when the TAR archive is created, depending on the commands the creator ran.
If the metadata file is not at the beginning of the file, that makes streaming completely inefficient. In this case the TAR archive needs to be recreated in an appropriate order.

Actually we could maybe just host the metadata file ourselves and then stream the audio data only. Don't think that this would be a problem for the FLEURS authors (I can ask them :-))

I made a PR to their repo to support streaming (by uploading the metadata file to the Hub). See:
- https://huggingface.co/datasets/google/fleurs/discussions/4

I'm closing this issue as the PR above has been merged.

## Dataset viewer issue for '*name of the dataset*'

https://huggingface.co/datasets/google/fleurs

```
Status code:   400
Exception:     NotImplementedError
Message:       Extraction protocol for TAR archives like 'https://storage.googleapis.com/xtreme_translations/FLEURS/af_za.tar.gz' is not implemented in streaming mode. Please use `dl_manager.iter_archive` instead.
```

Am I the one who added this dataset ? Yes

Can I fix this somehow in the script? @lhoestq @severo


Add some iteration method on a dataset column (specific for inference)

enhancement

New feature or request

Thanks for the suggestion ! I agree it would be nice to have something directly in `datasets` to do something as simple as that

cc @albertvillanova @mariosasko @polinaeterna What do you think if we have something similar to pandas `Series` that wouldn't bring everything in memory when doing `dataset["audio"]` ? Currently it returns a list with all the decoded audio data in memory.

It would be a breaking change though, since `isinstance(dataset["audio"], list)` wouldn't work anymore, but we could implement a `Sequence` so that `dataset["audio"][0]` still works and only loads one item in memory.

Your alternative suggestion with `iterate` is also sensible, though maybe less satisfactory in terms of experience IMO

I agree that current behavior (decoding all audio file sin the dataset when accessing `dataset["audio"]`) is not useful, IMHO. Indeed in our docs, we are constantly warning our collaborators not to do that.

Therefore I upvote for a "useful" behavior of `dataset["audio"]`. I don't think the breaking change is important in this case, as I guess no many people use it with its current behavior. Therefore, for me it seems reasonable to return a generator (instead of an in-memeory list) for "special" features, like Audio/Image.

@lhoestq on the other hand I don't understand your proposal about Pandas-like...

I recall I had the same idea while working on the `Image` feature, so I agree implementing something similar to `pd.Series` that lazily brings elements in memory would be beneficial.

@lhoestq @mariosasko Could you please give a link to that new feature of `pandas.Series`? As far as I remember since I worked with pandas for more than 6 years, there was no lazy in-memory feature; it was everything in-memory; that was the reason why other frameworks were created, like Vaex or Dask, e.g.

Yea pandas doesn't do lazy loading. I was referring to pandas.Series to say that they have a dedicated class to represent a column ;)

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

Currently, `dataset["audio"]` will load EVERY element in the dataset in RAM, which can be quite big for an audio dataset.
Having an iterator (or sequence) type of object, would make inference with `transformers` 's `pipeline` easier to use and not so memory hungry.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

For a non breaking change:

```python
for audio in dataset.iterate("audio"):
    # {"array": np.array(...), "sampling_rate":...}
```

For a  breaking change solution (not necessary), changing the type of `dataset["audio"]` to a sequence type so that

```python
pipe = pipeline(model="...")
for out in pipe(dataset["audio"]):
    # {"text":....}
```
could work

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

```python
def iterate(dataset, key):
    for item in dataset:
        yield dataset[key]

for out in pipeline(iterate(dataset, "audio")):
    # {"array": ...}
```

This works but requires the helper function which feels slightly clunky.

**Additional context**
Add any other context about the feature request here.

The context is actually to showcase better integration between  `pipeline` and `datasets` in the Quicktour demo: https://github.com/huggingface/transformers/pull/16723/files

@lhoestq