bug

Something isn't working

Fixed by installing either resampy<3 or resampy>=4

Hi, when i try to run prepare_dataset function in [fine tuning ASR tutorial 4](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb) , i got this error.
I got this error
Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.
There is no other logs available, so i have no clue what is the cause of it.
```

def prepare_dataset(batch):
    audio = batch["path"]
    # batched output is "un-batched"
    batch["input_values"] = processor(audio["array"], sampling_rate=audio["sampling_rate"]).input_values[0]
    batch["input_length"] = len(batch["input_values"])
    with processor.as_target_processor():
        batch["labels"] = processor(batch["text"]).input_ids
    return batch

data = data.map(prepare_dataset, remove_columns=data.column_names["train"],
                      num_proc=4)
```

Specify the actual results or traceback.
There is no traceback except
`Terminating: fork() called from a process already using GNU OpenMP, this is unsafe.`
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Add missing language tags to resources

_The documentation is not available anymore as the PR was closed or merged._

Add missing language tags to resources, required by existing datasets on GitHub.


Add add cc-by-sa-2.5 license tag

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4818). All of your documentation changes will be reflected on that endpoint.

- [ ] add it to moon-landing
- [ ] add it to hub-docs


Outdated Link for mkqa Dataset

Something isn't working

## Describe the bug
The URL used to download the mkqa dataset is outdated. It seems the URL to download the dataset is currently https://github.com/apple/ml-mkqa/blob/main/dataset/mkqa.jsonl.gz instead of https://github.com/apple/ml-mkqa/raw/master/dataset/mkqa.jsonl.gz (master branch has been renamed to main).

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("mkqa")
```

## Expected results
downloads the dataset

## Actual results
```python
Downloading builder script:
4.79k/? [00:00<00:00, 201kB/s]
Downloading metadata:
13.2k/? [00:00<00:00, 504kB/s]

Downloading and preparing dataset mkqa/mkqa (download: 11.35 MiB, generated: 34.29 MiB, post-processed: Unknown size, total: 45.65 MiB) to /home/lhr/.cache/huggingface/datasets/mkqa/mkqa/1.0.0/5401489c674c81257cf563417aaaa5de2c7e26a1090ce9b10eb0404f10003d4d...

Downloading data files: 0%
0/1 [00:00<?, ?it/s]

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Input In [3], in <cell line: 3>()
      1 from datasets import load_dataset
----> 3 dataset = load_dataset("mkqa")

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/load.py:1746, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1743 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1745 # Download and prepare data
-> 1746 builder_instance.download_and_prepare(
   1747     download_config=download_config,
   1748     download_mode=download_mode,
   1749     ignore_verifications=ignore_verifications,
   1750     try_from_hf_gcs=try_from_hf_gcs,
   1751     use_auth_token=use_auth_token,
   1752 )
   1754 # Build dataset for splits
   1755 keep_in_memory = (
   1756     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1757 )

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/builder.py:704, in DatasetBuilder.download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702         logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703 if not downloaded_from_gcs:
--> 704     self._download_and_prepare(
    705         dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706     )
    707 # Sync info
    708 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/builder.py:1227, in GeneratorBasedBuilder._download_and_prepare(self, dl_manager, verify_infos)
   1226 def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227     super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/builder.py:771, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    769 split_dict = SplitDict(dataset_name=self.name)
    770 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 771 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    773 # Checksums verification
    774 if verify_infos and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/mkqa/5401489c674c81257cf563417aaaa5de2c7e26a1090ce9b10eb0404f10003d4d/mkqa.py:130, in Mkqa._split_generators(self, dl_manager)
    128 # download and extract URLs
    129 urls_to_download = _URLS
--> 130 downloaded_files = dl_manager.download_and_extract(urls_to_download)
    132 return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={"filepath": downloaded_files["train"]})]

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/download/download_manager.py:431, in DownloadManager.download_and_extract(self, url_or_urls)
    415 def download_and_extract(self, url_or_urls):
    416     """Download and extract given url_or_urls.
    417
    418     Is roughly equivalent to:
   (...)
    429         extracted_path(s): `str`, extracted paths of given URL(s).
    430     """
--> 431     return self.extract(self.download(url_or_urls))

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/download/download_manager.py:309, in DownloadManager.download(self, url_or_urls)
    306 download_func = partial(self._download, download_config=download_config)
    308 start_time = datetime.now()
--> 309 downloaded_path_or_paths = map_nested(
    310     download_func,
    311     url_or_urls,
    312     map_tuple=True,
    313     num_proc=download_config.num_proc,
    314     disable_tqdm=not is_progress_bar_enabled(),
    315     desc="Downloading data files",
    316 )
    317 duration = datetime.now() - start_time
    318 logger.info(f"Downloading took {duration.total_seconds() // 60} min")

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py:393, in map_nested(function, data_struct, dict_only, map_list, map_tuple, map_numpy, num_proc, types, disable_tqdm, desc)
    391     num_proc = 1
    392 if num_proc <= 1 or len(iterable) <= num_proc:
--> 393     mapped = [
    394         _single_map_nested((function, obj, types, None, True, None))
    395         for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    396     ]
    397 else:
    398     split_kwds = []  # We organize the splits ourselve (contiguous splits)

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py:394, in <listcomp>(.0)
    391     num_proc = 1
    392 if num_proc <= 1 or len(iterable) <= num_proc:
    393     mapped = [
--> 394         _single_map_nested((function, obj, types, None, True, None))
    395         for obj in logging.tqdm(iterable, disable=disable_tqdm, desc=desc)
    396     ]
    397 else:
    398     split_kwds = []  # We organize the splits ourselve (contiguous splits)

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/utils/py_utils.py:330, in _single_map_nested(args)
    328 # Singleton first to spare some computation
    329 if not isinstance(data_struct, dict) and not isinstance(data_struct, types):
--> 330     return function(data_struct)
    332 # Reduce logging to keep things readable in multiprocessing with tqdm
    333 if rank is not None and logging.get_verbosity() < logging.WARNING:

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/download/download_manager.py:335, in DownloadManager._download(self, url_or_filename, download_config)
    332 if is_relative_path(url_or_filename):
    333     # append the relative path to the base_path
    334     url_or_filename = url_or_path_join(self._base_path, url_or_filename)
--> 335 return cached_path(url_or_filename, download_config=download_config)

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/utils/file_utils.py:185, in cached_path(url_or_filename, download_config, **download_kwargs)
    181     url_or_filename = str(url_or_filename)
    183 if is_remote_url(url_or_filename):
    184     # URL, so get it from the cache (downloading if necessary)
--> 185     output_path = get_from_cache(
    186         url_or_filename,
    187         cache_dir=cache_dir,
    188         force_download=download_config.force_download,
    189         proxies=download_config.proxies,
    190         resume_download=download_config.resume_download,
    191         user_agent=download_config.user_agent,
    192         local_files_only=download_config.local_files_only,
    193         use_etag=download_config.use_etag,
    194         max_retries=download_config.max_retries,
    195         use_auth_token=download_config.use_auth_token,
    196         ignore_url_params=download_config.ignore_url_params,
    197         download_desc=download_config.download_desc,
    198     )
    199 elif os.path.exists(url_or_filename):
    200     # File, and it exists.
    201     output_path = url_or_filename

File ~/repos/punc-cap/venv/lib/python3.9/site-packages/datasets/utils/file_utils.py:530, in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token, ignore_url_params, download_desc)
    525     raise FileNotFoundError(
    526         f"Cannot find the requested files in the cached path at {cache_path} and outgoing traffic has been"
    527         " disabled. To enable file online look-ups, set 'local_files_only' to False."
    528     )
    529 elif response is not None and response.status_code == 404:
--> 530     raise FileNotFoundError(f"Couldn't find file at {url}")
    531 _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    532 if head_error is not None:

FileNotFoundError: Couldn't find file at https://github.com/apple/ml-mkqa/raw/master/dataset/mkqa.jsonl.gz

```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.13.0-40-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 9.0.0
- Pandas version: 1.4.2


Update version of opus_paracrawl dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4816). All of your documentation changes will be reflected on that endpoint.

This PR updates OPUS ParaCrawl from 7.1 to 9 version.

Fix #4815.


Outdated loading script for OPUS ParaCrawl dataset

dataset bug

A bug in a dataset script provided in the library

## Describe the bug
Our loading script for OPUS ParaCrawl loads its 7.1 version. Current existing version is 9.


Support CSV as metadata file format in AudioFolder/ImageFolder

enhancement

New feature or request

Requested here: https://discuss.huggingface.co/t/how-to-structure-an-image-dataset-repo-using-the-image-folder-approach/21004. CSV is also used in AutoTrain for specifying metadata in image datasets.


Fix loading example in opus dataset cards

This PR:
- fixes the examples to load the datasets, with the corrected dataset name, in their dataset cards for:
  - opus_dgt
  - opus_paracrawl
  - opus_wikipedia
- fixes their dataset cards with the missing required information: title, data instances/fields/splits
- enumerates the supported languages
- adds a missing citation reference for opus_wikipedia

Related to:
- #4806


Fix bug in function validate_type for Python >= 3.9

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4812). All of your documentation changes will be reflected on that endpoint.

Fix `validate_type` function, so that it uses `get_origin` instead. This makes the function forward compatible.

This fixes #4811 because:
```python
In [4]: typing.Optional[str]
Out[4]: typing.Optional[str]

In [5]: get_origin(typing.Optional[str])
Out[5]: typing.Union
```

Fix #4811.


Bug in function validate_type for Python >= 3.9

bug

Something isn't working

## Describe the bug
The function `validate_type` assumes that the type `typing.Optional[str]` is automatically transformed to `typing.Union[str, NoneType]`.
```python
In [4]: typing.Optional[str]
Out[4]: typing.Union[str, NoneType]
```

However, this is not the case for Python 3.9:
```python
In [3]: typing.Optional[str]
Out[3]: typing.Optional[str]
```


Something isn't working


Add missing language tags to resources


Add add cc-by-sa-2.5 license tag


Outdated Link for mkqa Dataset

Something isn't working


enhancement

New feature or request


bug

Something isn't working


hellaswag: add non-empty description to fix metadata issue

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4810). All of your documentation changes will be reflected on that endpoint.


Complete the mlqa dataset card

> Thanks for your contribution, @eldhoittangeorge.
>
> The CI error message: https://github.com/huggingface/datasets/runs/7743526624?check_suite_focus=true
>
> ```
> E           ValueError: The following issues have been found in the dataset cards:
> E           YAML tags:
> E           __init__() missing 5 required positional arguments: 'annotations_creators', 'language_creators', 'license', 'size_categories', and 'source_datasets'
> ```

I will fix the CI error.

@eldhoittangeorge, thanks again for all the fixes. Just a minor one before we can merge this PR: https://github.com/huggingface/datasets/runs/7744885754?check_suite_focus=true
```
E           YAML tags:
E           Could not validate the metadata, found the following errors:
E           * field 'language_creators':
E           	['unknown'] are not registered tags for 'language_creators', reference at https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/creators.json
```

>

Thanks, I updated the file.
A small suggestion can you mention this link https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/  in the contribution page. So that others will know the acceptable values for the tags.

I fixed the issue #4808

Details of PR:
- Added languages included in the dataset.
- Added task id and task category.
- Updated the citation information.

Fix #4808.


Add more information to the dataset card of mlqa dataset


document fix in opus_gnome dataset

I fixed a issue #4805.

I changed `"gnome"` to `"opus_gnome"` in[ README.md](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary).


Fix opus_gnome dataset card

@gojiteji why have you closed this PR and created an identical one?
- #4807

@albertvillanova
I forgot to follow "How to create a Pull" in CONTRIBUTING.md in this branch.

Both are identical. And you can push additional commits to this branch.

Anyway, @gojiteji thanks for your contribution and this fix.

Once you have modified the `opus_gnome` dataset card, our Continuous Integration test suite performs some tests on it that make some additional requirements: the errors that appear have nothing to do with your contribution, but with these additional quality requirements.

> the errors that appear have nothing to do with your contribution, but with these additional quality requirements.

Is there anything I should do?

If you would like to address them as well in this PR, it would be awesome: https://github.com/huggingface/datasets/runs/7741104780?check_suite_focus=true

These are the 2 error messages:
```
E           ValueError: The following issues have been found in the dataset cards:
E           README Validation:
E           The following issues were found for the README at `/home/runner/work/datasets/datasets/datasets/opus_gnome/README.md`:
E           -	No first-level heading starting with `Dataset Card for` found in README. Skipping further validation for this README.

E           The following issues have been found in the dataset cards:
E           YAML tags:
E           Could not validate the metadata, found the following errors:
E           * field 'language':
E           	['ara', 'cat', 'foo', 'gr', 'nqo', 'tmp'] are not registered tags for 'language', reference at https://github.com/huggingface/datasets/tree/main/src/datasets/utils/resources/languages.json
```

In principle there are 2 errors:

The first one says, the title of the README does not start with `Dataset Card for`:
- The README title is: `# Dataset Card Creation Guide`
- According to the [template here](https://github.com/huggingface/datasets/blob/main/templates/README.md), it should be: `# Dataset Card for [Dataset Name]`

In relation with the languages:
- you should check whether the language codes are properly spelled
- and if so, adding them to our `languages.json` file, so that they are properly validated

Thank you for the detailed information. I'm checking it now.

```
E           ValueError: The following issues have been found in the dataset cards:
E           README Validation:
E           The following issues were found for the README at `/home/runner/work/datasets/datasets/datasets/opus_gnome/README.md`:
E           -	Expected some content in section `Data Instances` but it is empty.
E           -	Expected some content in section `Data Fields` but it is empty.
E           -	Expected some content in section `Data Splits` but it is empty.
```

I added `ara`, `cat`, `gr`, and `nqo` to  `languages.json` and removed `foo` and `tmp` from `README.md`.
I also write  Data Instances, Data Fields, and Data Splits in `README.md`.

Thanks for your investigation and fixes to the dataset card structure! I'm just making some suggestions before merging this PR: see below.

Should I create PR for `config.json` to add ` ara cat gr nqo` first?
I think I can pass this failing after that.

Or removing `ara, cat, gr, nqo, foo, tmp` from `README.md`.

Once you address these issues, all the CI tests will pass.

Once the remaining changes are addressed (see unresolved above), we will be able to merge this:
- [ ] Remove "ara" from README
- [ ] Remove "cat" from README
- [ ] Remove "gr" from README
- [ ] Replace "tmp" with "tyj" in README
- [ ] Add "tyj" to `languages.json`:
  ```
  "tyj": "Tai Do; Tai Yo",

I fixed a issue #4805.

I changed `"gnome"` to `"opus_gnome"` in[ README.md](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary).

Fix #4805


Wrong example in opus_gnome dataset card

Something isn't working

## Describe the bug
I found that [the example on opus_gone dataset ](https://github.com/huggingface/datasets/tree/main/datasets/opus_gnome#dataset-summary) doesn't work.

## Steps to reproduce the bug
```python
 load_dataset("gnome", lang1="it", lang2="pl")
```
`"gnome"` should be `"opus_gnome"`

## Expected results
```bash
100%
1/1 [00:00<00:00, 42.09it/s]
DatasetDict({
    train: Dataset({
        features: ['id', 'translation'],
        num_rows: 8368
    })
})
```

## Actual results
```bash
 Couldn't find 'gnome' on the Hugging Face Hub either: FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/main/datasets/gnome/gnome.py
```

## Environment info
- `datasets` version: 2.4.0
- Platform: Linux-5.4.0-120-generic-x86_64-with-glibc2.27
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3


Something isn't working

## Describe the bug
streaming dataset with concatenating splits raises an error

## Steps to reproduce the bug
```python
from datasets import load_dataset

# no error
repo = "nateraw/ade20k-tiny"
dataset = load_dataset(repo, split="train+validation")
```

```python
from datasets import load_dataset

# error
repo = "nateraw/ade20k-tiny"
dataset = load_dataset(repo, split="train+validation", streaming=True)
```

```sh
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-4-a6ae02d63899>](https://localhost:8080/#) in <module>()
      3 # error
      4 repo = "nateraw/ade20k-tiny"
----> 5 dataset = load_dataset(repo, split="train+validation", streaming=True)

1 frames
[/usr/local/lib/python3.7/dist-packages/datasets/builder.py](https://localhost:8080/#) in as_streaming_dataset(self, split, base_path)
   1030             splits_generator = splits_generators[split]
   1031         else:
-> 1032             raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
   1033
   1034         # Create a dataset for each of the given splits

ValueError: Bad split: train+validation. Available splits: ['validation', 'train']
```

[Colab](https://colab.research.google.com/drive/1wMj08_0bym9jnGgByib4lsBPu8NCZBG9?usp=sharing)

## Expected results
load successfully or throws an error saying it is not supported.

## Actual results
above

## Environment info
- `datasets` version: 2.4.0
- Platform: Windows-10-10.0.22000-SP0 (windows11 x64)
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Support `pipeline` argument in inspect.py functions

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**

The `wikipedia` dataset requires a `pipeline` argument to build the list of splits:

https://huggingface.co/datasets/wikipedia/blob/main/wikipedia.py#L937

But this is currently not supported in `get_dataset_config_info`:

https://github.com/huggingface/datasets/blob/main/src/datasets/inspect.py#L373-L375

which is called by other functions, e.g. `get_dataset_split_names`.

**Additional context**

The dataset viewer is not working out-of-the-box on `wikipedia` for this reason:

https://huggingface.co/datasets/wikipedia/viewer

<img width="637" alt="Capture d’écran 2022-08-08 à 12 01 16" src="https://user-images.githubusercontent.com/1676121/183461838-5330783b-0269-4ba7-a999-314cde2023d8.png">


`with_format` behavior is inconsistent on different datasets

bug

Something isn't working

Hi! You can get a `torch.Tensor` if you do the following:
```python
raw = load_dataset("beans", split="train")
raw = raw.select(range(100))

preprocessor = AutoFeatureExtractor.from_pretrained("nateraw/vit-base-beans")

from datasets import Array3D
features = raw.features.copy()
features["pixel_values"] = datasets.Array3D(shape=(3, 224, 224), dtype="float32")

def preprocess_func(examples):
    imgs = [img.convert("RGB") for img in examples["image"]]
    return preprocessor(imgs)

data = raw.map(preprocess_func, batched=True, features=features)

print(type(data[0]["pixel_values"]))

data = data.with_format("torch", columns=["pixel_values"])

print(type(data[0]["pixel_values"]))
```

The reason for this "inconsistency" in the default case is the way PyArrow infers the type of multi-dim arrays (in this case, the `pixel_values` column). If the type is not specified manually, PyArrow assumes it is a dynamic-length sequence (it needs to know the type before writing the first batch to a cache file, and it can't be sure the array is fixed ahead of time; `ArrayXD` is our way of telling that the dims are fixed), so it already fails to convert the corresponding array to NumPy properly (you get an array of `np.object` arrays). And `with_format("torch")` replaces NumPy arrays with Torch tensors, so this bad formatting propagates.

## Describe the bug
I found a case where `with_format` does not transform the dataset to the requested format.

## Steps to reproduce the bug
Run:

```python
from transformers import AutoTokenizer, AutoFeatureExtractor
from datasets import load_dataset

raw = load_dataset("glue", "sst2", split="train")
raw = raw.select(range(100))

tokenizer = AutoTokenizer.from_pretrained("philschmid/tiny-bert-sst2-distilled")

def preprocess_func(examples):
    return tokenizer(examples["sentence"], padding=True, max_length=256, truncation=True)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["input_ids"]))

data = data.with_format("torch", columns=["input_ids"])

print(type(data[0]["input_ids"]))
```

printing as expected:

```python
<class 'list'>
<class 'torch.Tensor'>
```

Then run:

```python
raw = load_dataset("beans", split="train")
raw = raw.select(range(100))

preprocessor = AutoFeatureExtractor.from_pretrained("nateraw/vit-base-beans")

def preprocess_func(examples):
    imgs = [img.convert("RGB") for img in examples["image"]]
    return preprocessor(imgs)

data = raw.map(preprocess_func, batched=True)

print(type(data[0]["pixel_values"]))

data = data.with_format("torch", columns=["pixel_values"])

print(type(data[0]["pixel_values"]))
```

Printing, unexpectedly

```python
<class 'list'>
<class 'list'>
```

## Expected results
`with_format` should transform into the requested format; it's not the case.

## Actual results
`type(data[0]["pixel_values"])` should be `torch.Tensor` in the example above

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: dev version, commit 44af3fafb527302282f6b6507b952de7435f0979
- Platform: Linux
- Python version: 3.9.12
- PyArrow version: 7.0.0


Fix fine classes in trec dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4801). All of your documentation changes will be reflected on that endpoint.

This PR:
- replaces the fine labels, so that there are 50 instead of 47
- once more labels are added, all they (fine and coarse) have been re-ordered, so that they align with the order in: https://cogcomp.seas.upenn.edu/Data/QA/QC/definition.html
- the feature names have been fixed: `fine_label` instead of `label-fine`
  - to sneak-case /underscores instead of hyphens)
  - words have been reordered

Fix #4790.


support LargeListArray in pyarrow

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4800). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Can you run `make style` at the repo root to fix the code quality error in CI and add a test?

Hi, I have fixed the code quality error and added a test

It seems that CI fails due to the lack of memory for allocating a large array, while I pass the test locally.

```python
import numpy as np
import datasets
a = np.zeros((5000000, 768))
res = datasets.Dataset.from_dict({"embedding": a})

'''
  File "/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/arrow_writer.py", line 178, in __arrow_array__
    out = numpy_to_pyarrow_listarray(data)
  File "/home/wenjiaxin/anaconda3/envs/data/lib/python3.8/site-packages/datasets/features/features.py", line 1173, in numpy_to_pyarrow_listarray
    offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32())
  File "pyarrow/array.pxi", line 312, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 83, in pyarrow.lib._ndarray_to_array
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 2147483904 not in range: -2147483648 to 2147483647
'''
```

Loading a large numpy array currently raises the error above as the type of offsets is `int32`.
And pyarrow has supported [LargeListArray](https://arrow.apache.org/docs/python/generated/pyarrow.LargeListArray.html) for this case.


video dataset loader/parser

enhancement

New feature or request

Hi! We've just started discussing the video support in `datasets` (decoding backends, video feature type, etc.), so I believe we should have something tangible by the end of this year.

Also, if you have additional video features in mind that you would like to see, feel free to let us know

you know how you can [use `load_dataset` with any arbitrary csv file](https://huggingface.co/docs/datasets/loading#csv)? and you can also [use it to load a local image dataset](https://huggingface.co/docs/datasets/image_load#local-files)?

could you please add functionality to load a video dataset? it would be really cool if i could point it to a bunch of video files and use pytorch to start looping through batches of videos. like if my batch size is 16, each sample in the batch is a frame from a video. i'm competing in the [minerl challenge](https://www.aicrowd.com/challenges/neurips-2022-minerl-basalt-competition), and it would be awesome to use the HF ecosystem.


Shard generator

Hi everyone! I was using Hugging Face datasets to process some very large datasets and found that it would be quite handy to have a feature that will allow to "split" these large datasets into chunks with equal size. Even better - be able to run through these chunks one by one in simple and convenient way. So I decided to add the method called shard_generator() to the main Dataset class. It works similar to shard method but it returns a generator of datasets with equal size (defined by shard_size attribute).
Example:
```python
>>> from datasets import load_dataset
>>> ds = load_dataset("rotten_tomatoes", split="validation")
>>> ds
Dataset({
      features: ['text', 'label'],
      num_rows: 1066
})
>>> next(ds.shard_generator(300))
Dataset({
      features: ['text', 'label'],
      num_rows: 300
})
```
I hope it can be helpful to someone. Thanks!


Torgo dataset creation

Hi @YingLi001, thanks for your proposal to add this dataset.

However, now we add datasets directly to the Hub (instead of our GitHub repository). You have the instructions in our docs:
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Create a dataset card](https://huggingface.co/docs/datasets/dataset_card)
- [Share](https://huggingface.co/docs/datasets/share)

Feel free to ask if you need any additional support/help.


ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB when adding image to Dataset

bug

Something isn't working

## Describe the bug

When adding a Pillow image to an existing Dataset on the hub, `add_item` fails due to the Pillow image not being automatically converted into the Image feature.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from PIL import Image

dataset = load_dataset("hf-internal-testing/example-documents")

# load any random Pillow image
image = Image.open("/content/cord_example.png").convert("RGB")

new_image = {'image': image}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Expected results
The image should be automatically casted to the Image feature when using `add_item`. For now, this can be fixed by using `encode_example`:

```
import datasets

feature = datasets.Image(decode=False)
new_image = {'image': feature.encode_example(image)}
dataset['test'] = dataset['test'].add_item(new_image)
```

## Actual results

```
ArrowInvalid: Could not convert <PIL.Image.Image image mode=RGB size=576x864 at 0x7F7CCC4589D0> with type Image: did not recognize Python value type when inferring an Arrow data type
```


MBPP splits

Something isn't working

Thanks for reporting this as well, @stadlerb.

I suggest waiting for the answer of the data owners...

(@albertvillanova)
The [MBPP dataset on the Hub](https://huggingface.co/datasets/mbpp) has only a test split for both its "full" and its "sanitized" subset, while the [paper](https://arxiv.org/abs/2108.07732) states in subsection 2.1 regarding the full split:
> In the experiments described later in the paper, we hold out 10 problems for **few-shot prompting**, another 500 as our **test** dataset (which is used to evaluate both few-shot inference and fine-tuned models), 374 problems for **fine-tuning**, and the rest for **validation**.

If the dataset on the Hub should reproduce most closely what the original authors use, I guess this four-way split should be reflected.

The paper doesn't explicitly state the task_id ranges of the splits, but the [GitHub readme](https://github.com/google-research/google-research/tree/master/mbpp) referenced in the paper specifies exact task_id ranges, although it misstates the total number of samples:
> We specify a train and test split to use for evaluation. Specifically:
>
> * Task IDs 11-510 are used for evaluation.
> * Task IDs 1-10 and 511-1000 are used for training and/or prompting. We typically used 1-10 for few-shot prompting, although you can feel free to use any of the training examples.

I.e. the few-shot, train and validation splits are combined into one split, with a soft suggestion of using the first ten for few-shot prompting. It is not explicitly stated whether the 374 fine-tuning samples mentioned in the paper have task_id 511 to 784 or 601 to 974 or are randomly sampled from task_id 511 to 974.

Regarding the "sanitized" split the paper states the following:
> For evaluations involving the edited dataset, we perform comparisons with 100 problems that appear in both the original and edited dataset, using the same held out 10 problems for few-shot prompting and 374 problems for fine-tuning.

The statement doesn't appear to be very precise, as among the 10 few-shot problems, those with task_id 1, 5 and 10 are not even part of the sanitized variant, and many from the task_id range from 511 to 974 are missing (e.g. task_id 511 to 553). I suppose the idea the task_id ranges for each split remain the same, even if some of the task_ids are not present. That would result in 7 few-shot, 257 test, 141 train and 22 validation examples in the sanitized split.


Add DocVQA

dataset request

Requesting to add a new dataset

Thanks for proposing, @NielsRogge.

Please, note this dataset requires registering in their website and their Terms and Conditions state we cannot distribute their URL:
```
1. You will NOT distribute the download URLs
...
```

## Adding a Dataset
- **Name:** DocVQA
- **Description:** Document Visual Question Answering (DocVQA) seeks to inspire a “purpose-driven” point of view in Document Analysis and Recognition research, where the document content is extracted and used to respond to high-level tasks defined by the human consumers of this information.
- **Paper:** https://arxiv.org/abs/2007.00398
- **Data:** https://www.docvqa.org/datasets/docvqa
- **Motivation:** Models like LayoutLM and Donut in the Transformers library are fine-tuned on DocVQA. Would be very handy to directly load this dataset from the hub.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Dataset Viewer issue for Team-PIXEL/rendered-wikipedia-english

dataset-viewer

Related to the dataset viewer on huggingface.co

Thanks for reporting. It's a known issue that should be fixed soon. Meanwhile, I had to manually trigger the dataset viewer. It's OK now.
Note that the extreme aspect ratio of the images generates another issue, that we're inspecting.

### Link

https://huggingface.co/datasets/Team-PIXEL/rendered-wikipedia-english/viewer/rendered-wikipedia-en/train

### Description

The dataset can be loaded fine but the viewer shows this error:

```
Server Error
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

I'm guessing this is because I recently renamed the dataset. Based on related issues (e.g. https://github.com/huggingface/datasets/issues/4759) , is there something server-side that needs to be refreshed?

### Owner

Yes


Issue with fine classes in trec dataset

bug

Something isn't working

## Describe the bug
According to their paper, the TREC dataset contains 2 kinds of classes:
- 6 coarse classes: TREC-6
- 50 fine classes: TREC-50

However, our implementation only has 47 (instead of 50) fine classes. The reason for this is that we only considered the last segment of the label, which is repeated for several coarse classes:
- We have one `desc` fine label instead of 2:
  - `DESC:desc`
  - `HUM:desc`
- We have one `other` fine label instead of 3:
  - `ENTY:other`
  - `LOC:other`
  - `NUM:other`

From their paper:
> We define a two-layered taxonomy, which represents a natural semantic classification for typical answers in the TREC task. The hierarchy contains 6 coarse classes and 50 fine classes,

> Each coarse class contains a non-overlapping set of fine classes.


Update doc upload_dataset.mdx

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4789). All of your documentation changes will be reflected on that endpoint.


Fix NonMatchingChecksumError in mbpp dataset

_The documentation is not available anymore as the PR was closed or merged._

Thank you for the quick response! Before noticing that you already had implemented the fix, I already had implemened my own version. I'd also suggest bumping the major version because the contents of the dataset changed, even if only slightly.
I'll attach my version of the affected files: [mbpp-checksum-changes.zip](https://github.com/huggingface/datasets/files/9258161/mbpp-checksum-changes.zip).

Hi @stadlerb, thanks for your feedback.

We normally update the major version whenever there is a new dataset release, usually with a breaking change in schema. The patch version is updated whenever there is a small correction in the dataset that does not change its schema.

As a side note for future contributions, please note that this dataset is hosted in our library GitHub repository. Therefore, the PRs to GitHub-hosted datasets needs being done through GitHub.

Currently added datasets are hosted on the Hub and for them, PRs can be done through the Hub.

I just noticed another problem with the dataset: The [GitHub page](https://github.com/google-research/google-research/tree/master/mbpp) and the [paper](http://arxiv.org/abs/2108.07732) mention a train-test split, which is not reflected in the dataloader. I'll open a new issue regarding this later.

Fix issue reported on the Hub: https://huggingface.co/datasets/mbpp/discussions/1

Fix #4787.


NonMatchingChecksumError in mbpp dataset

Something isn't working

## Describe the bug
As reported on the Hub [Fix Checksum Mismatch](https://huggingface.co/datasets/mbpp/discussions/1), there is a `NonMatchingChecksumError` when loading mbpp dataset

## Steps to reproduce the bug
```python
ds = load_dataset("mbpp", "full")
```

## Expected results
Loading of the dataset without any exception raised.

## Actual results
```
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-1-a3fbdd3ed82e> in <module>
----> 1 ds = load_dataset("mbpp", "full")

.../huggingface/datasets/src/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1791
   1792     # Download and prepare data
-> 1793     builder_instance.download_and_prepare(
   1794         download_config=download_config,
   1795         download_mode=download_mode,
.../huggingface/datasets/src/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, **download_and_prepare_kwargs)
    702                             logger.warning("HF google storage unreachable. Downloading and preparing it from source")
    703                     if not downloaded_from_gcs:
--> 704                         self._download_and_prepare(
    705                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    706                         )

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos)
   1225
   1226     def _download_and_prepare(self, dl_manager, verify_infos):
-> 1227         super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
   1228
   1229     def _get_examples_iterable_for_split(self, split_generator: SplitGenerator) -> ExamplesIterable:

.../huggingface/datasets/src/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    773         # Checksums verification
    774         if verify_infos and dl_manager.record_checksums:
--> 775             verify_checksums(
    776                 self.info.download_checksums, dl_manager.get_recorded_sizes_checksums(), "dataset source files"
    777             )

.../huggingface/datasets/src/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     38     if len(bad_urls) > 0:
     39         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 40         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     41     logger.info("All the checksums matched successfully" + for_verification_name)
     42

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://raw.githubusercontent.com/google-research/google-research/master/mbpp/mbpp.jsonl']
```


.save_to_disk('path', fs=s3) TypeError

Something isn't working

The following code:
```python
import datasets

train_dataset, test_dataset = load_dataset("imdb", split=["train", "test"])
s3 = datasets.filesystems.S3FileSystem(key=aws_access_key_id, secret=aws_secret_access_key)
train_dataset.save_to_disk("s3://datasets/", fs=s3)

```
produces following traceback:

```shell
File "C:\Users\Hong Knop\AppData\Local\Programs\Python\Python310\lib\site-packages\botocore\auth.py", line 374, in scope
    return '/'.join(scope)

```
I invoke print(scope) in <auth.py> (line 373) and find this:

```python
[('4VA08VLL3VTKQJKCAI8M',), '20220803', 'us-east-1', 's3', 'aws4_request']

```


Require torchaudio<0.12.0 in docs

This PR adds to docs the requirement of torchaudio<0.12.0 to avoid RuntimeError.

Subsequent to PR:
- #4777


Add Multiface dataset

dataset request

Requesting to add a new dataset

vision

Vision datasets

Hi @osanseviero I would like to add this dataset.

Hey @nandwalritik! Thanks for offering to help!

This dataset might be somewhat complex and I'm concerned about it being 65 TB, which would be quite expensive to host. @lhoestq @mariosasko I would love your input if you think it's worth adding this dataset.

Thanks for proposing this interesting dataset, @osanseviero.

Please note that the data files are already hosted in a third-party server: e.g. the index of data files for entity "6795937" is at https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/index.html
- audio.tar: https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/audio.tar
- ...

Therefore, in principle, we don't need to host them on our Hub: it would be enough to just implement a loading script in the corresponding Hub dataset repo, e.g. "facebook/multiface"...

## Adding a Dataset
- **Name:** Multiface dataset
- **Description:** f high quality recordings of the faces of 13 identities, each captured in a multi-view capture stage performing various facial expressions. An average of 12,200 (v1 scripts) to 23,000 (v2 scripts) frames per subject with capture rate at 30 fps
- **Data:** https://github.com/facebookresearch/multiface

The whole dataset is 65TB though, so I'm not sure

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


[WIP] Docs for creating a loading script for image datasets

documentation

Improvements or additions to documentation

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4783). All of your documentation changes will be reflected on that endpoint.

This PR is a first draft of creating a loading script for image datasets. Feel free to let me know if there are any specificities I'm missing for this. 🙂

To do:
- [x] Document how to create different configurations.


bug

Something isn't working

Thanks for reporting @conceptofmind.

Could you please give details about your environment?
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
```

Hi @albertvillanova ,

Here is the environment information:
```
- `datasets` version: 2.3.2
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.27
- Python version: 3.9.12
- PyArrow version: 7.0.0
- Pandas version: 1.4.2
```
Thanks,

Enrico

## Describe the bug
Following the example in CodeParrot, I receive an array size limitation error when deduplicating larger datasets.

## Steps to reproduce the bug
```python
dataset_name = "the_pile"
ds = load_dataset(dataset_name, split="train")
ds = ds.map(preprocess, num_proc=num_workers)
uniques = set(ds.unique("hash"))
```
Gists for minimum reproducible example:
https://gist.github.com/conceptofmind/c5804428ea1bd89767815f9cd5f02d9a
https://gist.github.com/conceptofmind/feafb07e236f28d79c2d4b28ffbdb6e2

## Expected results
Chunking and writing out a deduplicated dataset.

## Actual results
```
return dataset._data.column(column).unique().to_pylist()
File "pyarrow/table.pxi", line 394, in pyarrow.lib.ChunkedArray.unique
File "pyarrow/_compute.pyx", line 531, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 330, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 124, in pyarrow.lib.check_status
pyarrow.lib.ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483648
```


Fix label renaming and add a battery of tests

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4781). All of your documentation changes will be reflected on that endpoint.

This PR makes some changes to label renaming in `to_tf_dataset()`, both to fix some issues when users input something we weren't expecting, and also to make it easier to deprecate label renaming in future, if/when we want to move this special-casing logic to a function in `transformers`.

The main changes are:
- Label renaming now only happens when the `auto_rename_labels` argument is set. For backward compatibility, this defaults to `True` for now.
- If the user requests "label" but the data collator renames that column to "labels", the label renaming logic will now handle that case correctly.
- Added a battery of tests to make this more reliable in future.
- Adds an optimization to loading in `to_tf_dataset()` for unshuffled datasets (uses slicing instead of a list of indices)

Fixes #4772


Remove apache_beam import from module level in natural_questions dataset

Instead of importing `apache_beam` at the module level, import it in the method `_build_pcollection`.

Fix #4779.


Loading natural_questions requires apache_beam even with existing preprocessed data

Something isn't working

## Describe the bug
When loading "natural_questions", the package "apache_beam" is required:
```
ImportError: To be able to use natural_questions, you need to install the following dependency: apache_beam.
Please install it using 'pip install apache_beam' for instance'
```

This requirement is unnecessary, once there exists preprocessed data and the script just needs to download it.

## Steps to reproduce the bug
```python
load_dataset("natural_questions", "dev", split="validation", revision="main")
```

## Expected results
No ImportError raised.

## Actual results
```
ImportError                               Traceback (most recent call last)
[<ipython-input-3-c938e7c05d02>](https://localhost:8080/#) in <module>()
----> 1 from datasets import load_dataset; ds = load_dataset("natural_questions", "dev", split="validation", revision="main")

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1732         revision=revision,
   1733         use_auth_token=use_auth_token,
-> 1734         **config_kwargs,
   1735     )
   1736

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in load_dataset_builder(path, name, data_dir, data_files, cache_dir, features, download_config, download_mode, revision, use_auth_token, **config_kwargs)
   1504         download_mode=download_mode,
   1505         data_dir=data_dir,
-> 1506         data_files=data_files,
   1507     )
   1508

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1245                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"
   1246                     ) from None
-> 1247                 raise e1 from None
   1248     else:
   1249         raise FileNotFoundError(

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1180                     download_config=download_config,
   1181                     download_mode=download_mode,
-> 1182                     dynamic_modules_path=dynamic_modules_path,
   1183                 ).get_module()
   1184             elif path.count("/") == 1:  # community dataset on the Hub

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in get_module(self)
    490             base_path=hf_github_url(path=self.name, name="", revision=revision),
    491             imports=imports,
--> 492             download_config=self.download_config,
    493         )
    494         additional_files = [(config.DATASETDICT_INFOS_FILENAME, dataset_infos_path)] if dataset_infos_path else []

[/usr/local/lib/python3.7/dist-packages/datasets/load.py](https://localhost:8080/#) in _download_additional_modules(name, base_path, imports, download_config)
    214         _them_str = "them" if len(needs_to_be_installed) > 1 else "it"
    215         raise ImportError(
--> 216             f"To be able to use {name}, you need to install the following {_depencencies_str}: "
    217             f"{', '.join(needs_to_be_installed)}.\nPlease install {_them_str} using 'pip install "
    218             f"{' '.join(needs_to_be_installed.values())}' for instance'"

ImportError: To be able to use natural_questions, you need to install the following dependency: apache_beam.
Please install it using 'pip install apache_beam' for instance'
```

## Environment info
Colab notebook.


Update local loading script docs

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4778). All of your documentation changes will be reflected on that endpoint.

I would rather have a section in the docs that explains how to modify the script of an existing dataset (`inspect_dataset` + modification + `load_dataset`) instead of focusing on the GH datasets bundled with the source (only applicable for devs).

This PR clarifies the local loading script section to include how to load a dataset after you've modified the local loading script (closes #4732).


Require torchaudio<0.12.0 to avoid RuntimeError

Related to:
- https://github.com/huggingface/transformers/issues/18379

Fix partially #4776.


RuntimeError when using torchaudio 0.12.0 to load MP3 audio file

Requiring torchaudio<0.12.0 isn't really a viable solution because that implies torch<0.12.0 which means no sm_86 CUDA support which means no RTX 3090 support in PyTorch.

But in my case, the error only occurs if `_fallback_load` resolves to `_fail_load` inside torchaudio 0.12.0 which is only the case if FFMPEG initialization failed:  https://github.com/pytorch/audio/blob/b1f510fa5681e92ee82bdc6b2d1ed896799fc32c/torchaudio/backend/sox_io_backend.py#L36-L47

That means the proper solution for torchaudio>=0.12.0 is to check `torchaudio._extension._FFMPEG_INITIALIZED` and if it is False, then we need to remind the user to install a dynamically linked ffmpeg 4.1.8 and then maybe call `torchaudio._extension._init_ffmpeg()` to force a user-visible exception showing the missing ffmpeg dynamic library name.

On my system, installing

- libavcodec.so.58
- libavdevice.so.58
- libavfilter.so.7
- libavformat.so.58
- libavutil.so.56
- libswresample.so.3
- libswscale.so.5

from ffmpeg 4.1.8 made HF datasets 2.3.2 work just fine with  torchaudio 0.12.1+cu116:

```python3
import sox, torchaudio, datasets
print('torchaudio', torchaudio.__version__)
print('datasets', datasets.__version__)
torchaudio._extension._init_ffmpeg()
print(torchaudio._extension._FFMPEG_INITIALIZED)
waveform, sample_rate = torchaudio.load('/workspace/.cache/huggingface/datasets/downloads/extracted/8e5aa88585efa2a4c74c6664b576550d32b7ff9c3d1d17cc04f44f11338c3dc6/cv-corpus-8.0-2022-01-19/en/clips/common_voice_en_100038.mp3', format='mp3')
print(waveform.shape)
```

```
torchaudio 0.12.1+cu116
datasets 2.3.2
True
torch.Size([1, 369792])
```

Current version of `torchaudio` (0.12.0) raises a RuntimeError when trying to use `sox_io` backend but non-Python dependency `sox` is not installed:
https://github.com/pytorch/audio/blob/2e1388401c434011e9f044b40bc8374f2ddfc414/torchaudio/backend/sox_io_backend.py#L21-L29
```python
def _fail_load(
    filepath: str,
    frame_offset: int = 0,
    num_frames: int = -1,
    normalize: bool = True,
    channels_first: bool = True,
    format: Optional[str] = None,
) -> Tuple[torch.Tensor, int]:
    raise RuntimeError("Failed to load audio from {}".format(filepath))
```

Maybe we should raise a more actionable error message so that the user knows how to fix it.

UPDATE:
- this is an incompatibility of latest torchaudio (0.12.0) and the sox backend

TODO:
- [x] as a temporary solution, we should recommend installing torchaudio<0.12.0
  - #4777
  - #4785
- [ ] however, a stable solution must be found for torchaudio>=0.12.0

Related to:
- https://github.com/huggingface/transformers/issues/18379


Streaming not supported in Theivaprakasham/wildreceipt

Thanks for reporting @NitishkKarra.

The root source of the issue is that streaming mode is not supported out-of-the-box for that dataset, because it contains a TAR file.

We have opened a discussion in the corresponding Hub dataset page, pointing out this issue: https://huggingface.co/datasets/Theivaprakasham/wildreceipt/discussions/1

I'm closing this issue here, so this discussion is transferred there instead.

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Training hangs at the end of epoch, with set_transform/with_transform+multiple workers

Something isn't working

## Describe the bug
I use load_dataset() (I tried with [wiki](https://huggingface.co/datasets/wikipedia) and my own json data) and use set_transform/with_transform for preprocessing. But it hangs at the end of the 1st epoch if dataloader_num_workers>=1. No problem with single worker.

## Steps to reproduce the bug
```python
train_dataset = datasets.load_dataset("wikipedia", "20220301.en",
                                      split='train',
                                      cache_dir=model_args.cache_dir,
                                      streaming=False)
train_dataset.set_transform(psg_parse_fn)
train_dataloader = DataLoader(
            train_dataset,
            batch_size=args.train_batch_size,
            sampler=DistributedSampler(train_dataset),
            collate_fn=data_collator,
            drop_last=args.dataloader_drop_last,
            num_workers=args.dataloader_num_workers,
        )
```

## Expected results

## Actual results
It simply hangs. The ending step is num_example/batch_size (one epoch).

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Linux-5.4.170+-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 8.0.0
- Pandas version: 1.4.1


Document loading from relative path

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4773). All of your documentation changes will be reflected on that endpoint.

This PR describes loading a dataset from the Hub by specifying a relative path in `data_dir` or `data_files` in `load_dataset` (see #4757).


AssertionError when using label_cols in to_tf_dataset

Something isn't working

Hi @lehrig, this is caused by the data collator renaming "label" to "labels". If you set `label_cols=["labels"]` in the call it will work correctly. However, I agree that the cause of the bug is not obvious, so I'll see if I can make a PR to clarify things when the collator renames columns.

Thanks - and wow, that appears like a strange side-effect of the data collator. Is that really needed?

Why not make it more explicit? For example, extend `DefaultDataCollator` with an optional property `label_col_name` to be used as label column; only when it is not provided default to `labels` (and document that this happens) for backwards-compatibility?

Haha, I honestly have no idea why our data collators rename `"label"` (the standard label column name in our datasets) to `"labels"` (the standard label column name input to our models). It's been a pain point when I design TF data pipelines, though, because I don't want to hardcode things like that - especially in `datasets`, because the renaming is something that happens purely at the `transformers` end. I don't think I could make the change in the data collators themselves at this point, because it would break backward compatibility for everything in PyTorch as well as TF.

In the most recent version of `transformers` we added a [prepare_tf_dataset](https://huggingface.co/docs/transformers/main_classes/model#transformers.TFPreTrainedModel.prepare_tf_dataset) method to our models which takes care of these details for you, and even chooses appropriate columns and labels for the model you're using. In future we might make that the officially recommended way to convert HF datasets to `tf.data.Dataset`.

Interesting, that'd be great especially for clarity. https://huggingface.co/docs/datasets/use_with_tensorflow#data-loading already improved clarity, yet, all those options will still confuse people. Looking forward to those advances in the hope there'll be only 1 way in the future ;)

Anyways, I am happy for the time being with the work-around you provided. Thank you!

## Describe the bug
An incorrect `AssertionError` is raised when using `label_cols` in `to_tf_dataset` and the label's key name is `label`.

The assertion is in this line:
https://github.com/huggingface/datasets/blob/2.4.0/src/datasets/arrow_dataset.py#L475

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import DefaultDataCollator

dataset = load_dataset('glue', 'mrpc', split='train')

tf_dataset = dataset.to_tf_dataset(
    columns=["sentence1", "sentence2", "idx"],
    label_cols=["label"],
    batch_size=16,
    collate_fn=DefaultDataCollator(return_tensors="tf"),
)
```

## Expected results
No assertion error.

## Actual results
```
AssertionError: in user code:

    File "/opt/conda/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 475, in split_features_and_labels  *
        assert set(features.keys()).union(labels.keys()) == set(input_batch.keys())
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.18.0-305.45.1.el8_4.ppc64le-ppc64le-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 7.0.0
- Pandas version: 1.4.3


Remove dummy data generation docs

_The documentation is not available anymore as the PR was closed or merged._

This PR removes instructions to generate dummy data since that is no longer necessary for datasets that are uploaded to the Hub instead of our GitHub repo.

Close #4744


fix typo

> good catch thanks ! Can you check if the same typo is also present in `add_elasticsearch_index` ? It has a very similar signature

fixed

By defaul -> By default


Fail to process SQuADv1.1 datasets with max_seq_length=128, doc_stride=96.

Something isn't working

## Describe the bug
datasets fail to process SQuADv1.1 with max_seq_length=128, doc_stride=96 when calling datasets["train"].train_dataset.map().

## Steps to reproduce the bug

I used huggingface[ TF2 question-answering examples](https://github.com/huggingface/transformers/tree/main/examples/tensorflow/question-answering). And my scripts are as follows:

```
python run_qa.py \
  --model_name_or_path $BERT_DIR \
  --dataset_name $SQUAD_DIR \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 128 \
  --doc_stride 96 \
  --output_dir $OUTPUT \
  --save_steps 10000 \
  --overwrite_cache \
  --overwrite_output_dir \

```

## Expected results
Normally process SQuADv1.1 datasets with max_seq_length=128, doc_stride=96.

## Actual results
```
INFO:__main__:Padding all batches to max length because argument was set or we're on TPU.
WARNING:datasets.fingerprint:Parameter 'function'=<function main.<locals>.prepare_train_features at 0x7f15bc2d07a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.
  0%|                                                                                | 0/88 [00:00<?, ?ba/s]thread '<unnamed>' panicked at 'assertion failed: stride < max_len', /__w/tokenizers/tokenizers/tokenizers/src/tokenizer/encoding.rs:311:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
  0%|                                                                                | 0/88 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "run_qa.py", line 743, in <module>
    main()
  File "run_qa.py", line 485, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2394, in map
    desc=desc,
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 551, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 518, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/anaconda3/envs/py37/lib/python3.7/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2768, in _map_single
    offset=offset,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2644, in apply_function_on_filtered_inputs
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
  File "anaconda3/envs/py37/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 2336, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "run_qa.py", line 410, in prepare_train_features
    padding=padding,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2512, in __call__
    **kwargs,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_base.py", line 2703, in batch_encode_plus
    **kwargs,
  File "anaconda3/envs/py37/lib/python3.7/site-packages/transformers/tokenization_utils_fast.py", line 429, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
pyo3_runtime.PanicException: assertion failed: stride < max_len
Traceback (most recent call last):
  File "./data/SQuADv1.1/evaluate-v1.1.py", line 92, in <module>
    with open(args.prediction_file) as prediction_file:
FileNotFoundError: [Errno 2] No such file or directory: './output/bert_base_squadv1.1_tf2/eval_predictions.json'

```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Ubuntu, pytorch=1.11.0, tensorflow-gpu=2.9.1
- Python version: 2.7
- PyArrow version: 8.0.0


Unpin rouge_score test dependency

Once `rouge-score` has made the 0.1.2 release to fix their issue https://github.com/google-research/google-research/issues/1212, we can unpin it.

Related to:
- #4735


Add 2.4.0 version added to docstrings


Dataset Viewer issue for openclimatefix/goes-mrms

Thanks for reporting, @cheaterHy.

The cause of this issue is a misalignment between the names of the repo (`goes-mrms`, with hyphen) and its Python loading scrip file (`goes_mrms.py`, with underscore).

I've opened an Issue discussion in their repo: https://huggingface.co/datasets/openclimatefix/goes-mrms/discussions/1


Fix version in map_nested docstring

After latest release, `map_nested` docstring needs being updated with the right version for versionchanged and versionadded.


Update CI badge

Replace the old CircleCI badge with a new one for GH Actions.


More rigorous shape inference in to_tf_dataset

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4763). All of your documentation changes will be reflected on that endpoint.

`tf.data` needs to know the shape of tensors emitted from a `tf.data.Dataset`. Although `None` dimensions are possible, overusing them can cause problems - Keras uses the dataset tensor spec at compile-time, and so saying that a dimension is `None` when it's actually constant can hurt performance, or even cause training to fail for dimensions that are needed to determine the shape of weight tensors!

The compromise I used here was to sample several batches from the underlying dataset and apply the `collate_fn` to them, and then to see which dimensions were "empirically variable". There's an obvious problem here, though - if you sample 10 batches and they all have the same shape on a certain dimension, there's still a small chance that the 11th batch will be different, and Keras will throw an error  if a dataset tries to emit a tensor whose shape doesn't match the spec.

I encountered this bug in practice once or twice for datasets that were mostly-but-not-totally constant on a given dimension, and I still don't have a perfect solution, but this PR should greatly reduce the risk. It samples many more batches, and also samples very small batches (size 2) - this increases the variability, making it more likely that a few outlier samples will be detected.

Ideally, of course, we'd determine the full output shape analytically, but that's surprisingly tricky when the `collate_fn` can be any arbitrary Python code!


Improve features resolution in streaming

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4762). All of your documentation changes will be reflected on that endpoint.

`IterableDataset._resolve_features` was returning the features sorted alphabetically by column name, which is not consistent with non-streaming. I changed this and used the order of columns from the data themselves. It was causing some inconsistencies in the dataset viewer as well.

I also fixed `interleave_datasets` that was not filling missing columns with None, because it was not using the columns from `IterableDataset._resolve_features`

cc @severo


parallel searching in multi-gpu setting using faiss

And I don't see any speed up when increasing the number of GPUs while calling `get_nearest_examples_batch`.

Hi ! Yes search_batch uses FAISS search which happens in parallel across the GPUs

> And I don't see any speed up when increasing the number of GPUs while calling get_nearest_examples_batch.

That's unexpected, can you share the code you're running ?

here is the code snippet

```python

# add faiss index
source_dataset = load_dataset(source_path)
queries = load_dataset(query_path)
gpu = [0,1,2,3]
source_dataset.add_faiss_index(
                "embedding",
                device=gpu,
            )

# batch query
batch_size = 32
for i in tqdm(range(0, len(queries), batch_size)):
    if i + batch_size >= len(queries):
        batched_queries = queries[i:]
    else:
        batched_queries = queries[i:i+batch_size]

    batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
    scores, candidates = source_dataset.get_nearest_examples_batch(
        "embedding",
        batched_query_embeddings,
        k=5
    )
```

The code looks all good to me, do you see all the GPUs being utilized ? What version of faiss are you using ?

I can see the memory usage of all the GPUs.
My version of `faiss-gpu` is `1.7.2`

It looks all good to me then ^^ though you said you didn't experienced speed improvements by adding more GPUs ? What size is your source dataset and what time differences did you experience ?

query set: 1e6
source dataset: 1e6
embedding size: 768
index: Flat
topk: 20
GPU: V100

The time taken to traverse the query set once is about 1.5h, which is almost not influenced by the value of query batch size or the number of GPUs according to my experiments.

Hmmm the number of GPUs should divide the time, something is going wrong. Can you check that adding more GPU does divide the memory used per GPU ? Maybe it can be worth looking at similar issues in the FAISS repository or create a noew issue over there to understand what's going on

> Can you check that adding more GPU does divide the memory used per GPU

The memory used per GPU is unchanged while adding more GPU. Is this unexpected?

I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆

> I used to think that every GPU loads all the source vectors and the data parallelism is at the query level. 😆

Oh indeed that's possible, I wasn't sure. Anyway you can check that calling get_nearest_examples_batch simply calls search under the hood:

https://github.com/huggingface/datasets/blob/f90f71fbbb33889fe75a3ffc101cdf16a88a3453/src/datasets/search.py#L375

Here is a runnable script.
Multi-GPU searching still does not work in my experiments.

```python
import os
from tqdm import tqdm
import numpy as np
import datasets
from datasets import Dataset

class DPRSelector:

    def __init__(self, source, target, index_name, gpu=None):
        self.source = source
        self.target = target
        self.index_name = index_name

        cache_path = 'embedding.faiss'

        if not os.path.exists(cache_path):
            self.source.add_faiss_index(
                column="embedding",
                index_name=index_name,
                device=gpu,
            )
            self.source.save_faiss_index(index_name, cache_path)
        else:
            self.source.load_faiss_index(
                index_name,
                cache_path,
                device=gpu
            )
        print('index builded!')

    def build_dataset(self, top_k, batch_size):
        print('start search')

        for i in tqdm(range(0, len(self.target), batch_size)):
            if i + batch_size >= len(self.target):
                batched_queries = self.target[i:]
            else:
                batched_queries = self.target[i:i+batch_size]

            batched_query_embeddings = np.stack([i for i in batched_queries['embedding']], axis=0)
            search_res = self.source.get_nearest_examples_batch(
                self.index_name,
                batched_query_embeddings,
                k=top_k
            )

        print('finish search')

def get_pseudo_dataset():
    pseudo_dict = {"embedding": np.zeros((1000000, 768), dtype=np.float32)}
    print('generate pseudo data')

    dataset = Dataset.from_dict(pseudo_dict)
    def list_to_array(data):
        return {"embedding": [np.array(vector, dtype=np.float32) for vector in data["embedding"]]}
    dataset.set_transform(list_to_array, columns='embedding', output_all_columns=True)

    print('build dataset')
    return dataset

if __name__=="__main__":

    np.random.seed(42)

    source_dataset = get_pseudo_dataset()
    target_dataset = get_pseudo_dataset()

    gpu = [0,1,2,3,4,5,6,7]
    selector = DPRSelector(source_dataset, target_dataset, "embedding",  gpu=gpu)

    selector.build_dataset(top_k=20, batch_size=32)
```

@lhoestq Hi, could you please test the code above if you have time? 😄

Maybe @albertvillanova you can take a look ? I won't be available in the following days

@albertvillanova Hi, can you help with this issue?

Hi @xwwwwww I'm investigating it, but I'm not an expert in Faiss. In principle, it is weird that your code does not work properly because it seems right...

> Have you tried passing `gpu=-1` and check if there is a speedup?

yes, there is a speed up using GPU compared with CPU.

When passing `device=-1`, ALL existing GPUs are used (multi GPU): this is the maximum speedup you can get. To know the number of total GPUs:
```
import faiss

ngpus = faiss.get_num_gpus()
print(ngpus)
```

When passing a list of integers to `device`, then only that number of GPUs are used (multi GPU as well)
- the speedup should be proportional (more or less) to the ratio of the number of elements passed to `device` over `ngpus`
- if this is not the case, then there is an issue in the implementation of this use case (however, I have reviewed the code and in principle I can't find any evident bug)

When passing a positive integer to `device`, then only a single GPU is used.
- this time should be more or less proportional to the time when passing `device=-1` over `ngpus`

Thanks for your help!
Have you run the code and replicated the same experimental results (i.e., no speedup while increasing the number of GPUs)?

@albertvillanova @lhoestq Sorry for the bother, is there any progress on this issue? 😃

While I notice that  `add_faiss_index` has supported assigning multiple GPUs, I am still confused about how it works.

Does the `search-batch` function automatically parallelizes the input queries to different gpus?https://github.com/huggingface/datasets/blob/d76599bdd4d186b2e7c4f468b05766016055a0a5/src/datasets/search.py#L360


Issue with offline mode

Something isn't working

Hi @SaulLu, thanks for reporting.

I think offline mode is not supported for datasets containing only data files (without any loading script). I'm having a look into this...

Thanks for your feedback!

To give you a little more info, if you don't set the offline mode flag, the script will load the cache. I first noticed this behavior with the `evaluate` library, and while trying to understand the downloading flow I realized that I had a similar error with datasets.

This is related to https://github.com/huggingface/datasets/issues/3547

## Describe the bug
I can't retrieve a cached dataset with offline mode enabled

## Steps to reproduce the bug

To reproduce my issue, first, you'll need to run a script that will cache the dataset
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "0"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```
then, you can try to reload it in offline mode:
```python
import os
os.environ["HF_DATASETS_OFFLINE"] = "1"

import datasets

datasets.logging.set_verbosity_info()
ds_name = "SaulLu/toy_struc_dataset"
ds = datasets.load_dataset(ds_name)
print(ds)
```

## Expected results
I would have expected the 2nd snippet not to return any errors

## Actual results
The 2nd snippet returns:
```
Traceback (most recent call last):
  File "/home/lucile_huggingface_co/sandbox/evaluate/test_cache_datasets.py", line 8, in <module>
    ds = datasets.load_dataset(ds_name)
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1723, in load_dataset
    builder_instance = load_dataset_builder(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1500, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/home/lucile_huggingface_co/anaconda3/envs/evaluate-dev/lib/python3.8/site-packages/datasets/load.py", line 1241, in dataset_module_factory
    raise ConnectionError(f"Couln't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'SaulLu/toy_struc_dataset': Offline mode is enabled.
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: Linux-4.19.0-21-cloud-amd64-x86_64-with-glibc2.17
- Python version: 3.8.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.3

Maybe I'm misunderstanding something in the use of the offline mode (see [doc](https://huggingface.co/docs/datasets/v2.4.0/en/loading#offline)), is that the case?


Dataset Viewer issue for Toygar/turkish-offensive-language-detection

I refreshed the dataset viewer manually, it's fixed now. Sorry for the inconvenience.
<img width="1557" alt="Capture d’écran 2022-07-28 à 09 17 39" src="https://user-images.githubusercontent.com/1676121/181514666-92d7f8e1-ddc1-4769-84f3-f1edfdb902e8.png">

### Link

https://huggingface.co/datasets/Toygar/turkish-offensive-language-detection

### Description

Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.

Hi, I provided train.csv, test.csv and valid.csv files. However, viewer says dataset does not exist.
Should I need to do anything else?

### Owner

Yes


Document better when relative paths are transformed to URLs

As discussed with @ydshieh, when passing a relative path as `data_dir` to `load_dataset` of a dataset hosted on the Hub, the relative path is transformed to the corresponding URL of the Hub dataset.

Currently, we mention this in our docs here: [Create a dataset loading script > Download data files and organize splits](https://huggingface.co/docs/datasets/v2.4.0/en/dataset_script#download-data-files-and-organize-splits)
> If the data files live in the same folder or repository of the dataset script, you can just pass the relative paths to the files instead of URLs.

Maybe we should document better how relative paths are handled, not only when creating a dataset loading script, but also when passing to `load_dataset`:
- `data_dir`
- `data_files`

CC: @stevhliu


Datasets.map causes incorrect overflow_to_sample_mapping when used with tokenizers and small batch size

Something isn't working

I've built a minimal example that shows this bug without `n_proc`. It seems like it's a problem any way of using **tokenizers, `overflow_to_sample_mapping`, and Dataset.map, with a small batch size**:

```
import datasets
import transformers
pretrained = 'deepset/tinyroberta-squad2'
tokenizer = transformers.AutoTokenizer.from_pretrained(pretrained)

questions = ['Can you tell me why?', 'What time is it?']
contexts = ['This is context zero', 'Another paragraph goes here']

def tok(questions, contexts):
    return tokenizer(text=questions,
            text_pair=contexts,
            truncation='only_second',
            return_overflowing_tokens=True,
            )
print(tok(questions, contexts)['overflow_to_sample_mapping'])
assert tok(questions, contexts)['overflow_to_sample_mapping'] == [0, 1] # PASSES

def tok2(d):
    return tok(d['question'], d['context'])

def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=1)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # FAILS produces [0,0]
```

Note that even if the batch size would be larger, there will be instances where we will not have a lot of data, and end up using small batches.  This can occur e.g. if `n_proc` causes batches to be underfill.  I imagine it can also occur in other ways, e.g. the final leftover batch at the end.

A larger batch size does _not_ have this behavior:

```
def tok2(d):
    return tok(d['question'], d['context'])

ds = datasets.Dataset.from_dict({'question': questions, 'context': contexts})
tokens = ds.map(tok2, batched=True, batch_size=2)
print(tokens['overflow_to_sample_mapping'])
assert tokens['overflow_to_sample_mapping'] == [0, 1] # PASSES
```

## Describe the bug
When using `tokenizer`, we can retrieve the field `overflow_to_sample_mapping`, since long samples will be overflown into multiple token sequences.

However, when tokenizing is done via `Dataset.map`, with `n_proc > 1`, the `overflow_to_sample_mapping` field is wrong.  This seems to be because each tokenizer only looks at its share of the samples, and maps to the index _within its share_, but then `Dataset.map` collates them together.

## Steps to reproduce the bug

1. Make a dataset of 3 strings.
2. Tokenize via Dataset.map with n_proc = 8
3. Inspect the `overflow_to_sample_mapping` field

## Expected results
`[0, 1, 2]`

## Actual results
`[0, 0, 0]`

Notes:

1. I have not yet extracted a minimal example, but the above works reliably
2. If the dataset is large, I've yet to determine if this bug still happens a. not at all b. always c. on the small, leftover batch at the end.


Remove "unkown" language tags

_The documentation is not available anymore as the PR was closed or merged._

Following https://github.com/huggingface/datasets/pull/4753 there was still a "unknown" langauge tag in `wikipedia` so the job at https://github.com/huggingface/datasets/runs/7542567336?check_suite_focus=true failed for wikipedia


Add `language_bcp47` tag

Following (internal) https://github.com/huggingface/moon-landing/pull/3509, we need to move the bcp47 tags to `language_bcp47` and keep the `language` tag for iso 639 1-2-3 codes. In particular I made sure that all the tags in `languages` are not longer than 3 characters. I moved the rest to `language_bcp47` and fixed some of them.

After this PR is merged I think we can simplify the language validation from the DatasetMetadata class (and keep it bare-bone just for the tagging app)

PS: the CI is failing because of missing content in dataset cards that are unrelated to this PR


DatasetInfo issue when testing multiple configs: mixed task_templates

Something isn't working

I've narrowed down the issue to the `dataset_module_factory` which already creates a `dataset_infos.json` file down in the `.cache/modules/dataset_modules/..` folder. That JSON file already contains the wrong task_templates for `unfiltered`.

Ugh. Found the issue: apparently `datasets` was reusing the already existing `dataset_infos.json` that is inside `datasets/datasets/hebban-reviews`! Is this desired behavior?

Perhaps when `--save_infos` and `--all_configs` are given, an existing `dataset_infos.json` file should first be deleted before continuing with the test? Because that would assume that the user wants to create a new infos file for all configs anyway.

Hi! I think this is a reasonable solution. Would you be interested in submitting a PR?

## Describe the bug
When running the `datasets-cli test` it would seem that some config properties in a DatasetInfo get mangled, leading to issues, e.g., about the ClassLabel.

## Steps to reproduce the bug

In summary, what I want to do is create three configs:
- unfiltered: no classlabel, no tasks. Gets data from unfiltered.json.gz (I'd want this without splits, just one chunk of data, but that does not seem possible?)
- filtered_sentiment: `review_sentiment` as ClassLabel, TextClassification task with `review_sentiment` as label. Gets train/test split from respective json.gz files
- filtered_rating: `review_rating0` as ClassLabel, TextClassification task with `review_rating0` as label. Gets train/test split from respective json.gz files

This might be a bit tedious to reproduce, so I am sorry, but these are the steps:

- Clone datasets -> `datasets/` and install it
- Clone `https://huggingface.co/datasets/BramVanroy/hebban-reviews` into `datasets/datasets` so that you have a new folder `datasets/datasets/hebban-reviews/`.
- Replace the HebbanReviews class with this new one:

```python
class HebbanReviews(datasets.GeneratorBasedBuilder):
    """The Hebban book reviews dataset."""

    BUILDER_CONFIGS = [
        HebbanReviewsConfig(
            name="unfiltered",
            description=_HEBBAN_REVIEWS_UNFILTERED_DESCRIPTION,
            version=datasets.Version(_HEBBAN_VERSION)
        ),
        HebbanReviewsConfig(
            name="filtered_sentiment",
            description=f"This config has the negative, neutral, and positive sentiment scores as ClassLabel in the 'review_sentiment' column.\n{_HEBBAN_REVIEWS_FILTERED_DESCRIPTION}",
            version=datasets.Version(_HEBBAN_VERSION)
        ),
        HebbanReviewsConfig(
            name="filtered_rating",
            description=f"This config has the 5-class ratings as ClassLabel in the 'review_rating0' column (which is a variant of 'review_rating' that starts counting from 0 instead of 1).\n{_HEBBAN_REVIEWS_FILTERED_DESCRIPTION}",
            version=datasets.Version(_HEBBAN_VERSION)
        )
    ]

    DEFAULT_CONFIG_NAME = "filtered_sentiment"

    _URLS = {
        "train": "train.jsonl.gz",
        "test": "test.jsonl.gz",
        "unfiltered": "unfiltered.jsonl.gz",
    }

    def _info(self):
        features = {
            "review_title": datasets.Value("string"),
            "review_text": datasets.Value("string"),
            "review_text_without_quotes": datasets.Value("string"),
            "review_n_quotes": datasets.Value("int32"),
            "review_n_tokens": datasets.Value("int32"),
            "review_rating": datasets.Value("int32"),
            "review_rating0": datasets.Value("int32"),
            "review_author_url": datasets.Value("string"),
            "review_author_type": datasets.Value("string"),
            "review_n_likes": datasets.Value("int32"),
            "review_n_comments": datasets.Value("int32"),
            "review_url": datasets.Value("string"),
            "review_published_date": datasets.Value("string"),
            "review_crawl_date": datasets.Value("string"),
            "lid": datasets.Value("string"),
            "lid_probability": datasets.Value("float32"),
            "review_sentiment": datasets.features.ClassLabel(names=["negative", "neutral", "positive"]),
            "review_sentiment_label": datasets.Value("string"),
            "book_id": datasets.Value("int32"),
        }

        if self.config.name == "filtered_sentiment":
            task_templates = [datasets.TextClassification(text_column="review_text_without_quotes", label_column="review_sentiment")]
        elif self.config.name == "filtered_rating":
            # For CrossEntropy, our classes need to start at index 0 -- not 1
            features["review_rating0"] = datasets.features.ClassLabel(names=["1", "2", "3", "4", "5"])
            features["review_sentiment"] = datasets.Value("int32")
            task_templates = [datasets.TextClassification(text_column="review_text_without_quotes", label_column="review_rating0")]
        elif self.config.name == "unfiltered":  # no ClassLabels in unfiltered
            features["review_sentiment"] = datasets.Value("int32")
            task_templates = None
        else:
            raise ValueError(f"Unsupported config {self.config.name}. Expected one of 'filtered_sentiment' (default),"
                             f" 'filtered_rating', or 'unfiltered'")
        print("AT INFO", self.config.name, task_templates)
        return datasets.DatasetInfo(
            description=self.config.description,
            features=datasets.Features(features),
            homepage="https://huggingface.co/datasets/BramVanroy/hebban-reviews",
            citation=_HEBBAN_REVIEWS_CITATION,
            task_templates=task_templates,
            license="cc-by-4.0"
        )

    def _split_generators(self, dl_manager):
        if self.config.name.startswith("filtered"):
            files = dl_manager.download_and_extract({"train": "train.jsonl.gz",
                                                     "test": "test.jsonl.gz"})
            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "data_file": files["train"]
                    },
                ),
                datasets.SplitGenerator(
                    name=datasets.Split.TEST,
                    gen_kwargs={
                        "data_file": files["test"]
                    },
                ),
            ]
        elif self.config.name == "unfiltered":
            files = dl_manager.download_and_extract({"train": "unfiltered.jsonl.gz"})
            return [
                datasets.SplitGenerator(
                    name=datasets.Split.TRAIN,
                    gen_kwargs={
                        "data_file": files["train"]
                    },
                ),
            ]
        else:
            raise ValueError(f"Unsupported config {self.config.name}. Expected one of 'filtered_sentiment' (default),"
                             f" 'filtered_rating', or 'unfiltered'")

    def _generate_examples(self, data_file):
        lines = Path(data_file).open(encoding="utf-8").readlines()
        for line_idx, line in enumerate(lines):
            row = json.loads(line)
            yield line_idx, row
```

- finally, run `datasets-cli test ./datasets/hebban-reviews/ --save_infos --all_configs` from within the topmost `datasets` directory

## Expected results
Succeeding tests for three different configs.

## Actual results

I printed out the values that are given to `DatasetInfo` for config name and task_templates, as you can see. There, as expected, I get `unfiltered None`. I also modified datasets/info.py and added this line [at L.170](https://github.com/huggingface/datasets/blob/f5847a304aa1b38b3a3c54a8318b4df60f1299bc/src/datasets/info.py#L170):

```python
print("INTERNALLY AT INFO.PY", self.config_name, self.task_templates)
```

to my surprise, here I get `unfiltered [TextClassification(task='text-classification', text_column='review_text_without_quotes', label_column='review_sentiment')]`. So one way or another, here I suddenly see that `unfiltered` now does have a task_template -- even though that is not what is written in the data loading script, as the first print statement correctly shows.

I do not quite understand how, but it seems that the config name and task_templates get mixed.

This ultimately leads to the following error, but this trace may not be very useful in itself:

```
Traceback (most recent call last):
  File "C:\Users\bramv\.virtualenvs\hebban-U6poXNQd\Scripts\datasets-cli-script.py", line 33, in <module>
    sys.exit(load_entry_point('datasets', 'console_scripts', 'datasets-cli')())
  File "c:\dev\python\hebban\datasets\src\datasets\commands\datasets_cli.py", line 39, in main
    service.run()
  File "c:\dev\python\hebban\datasets\src\datasets\commands\test.py", line 144, in run
    builder.as_dataset()
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 899, in as_dataset
    datasets = map_nested(
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 393, in map_nested
    mapped = [
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 394, in <listcomp>
    _single_map_nested((function, obj, types, None, True, None))
  File "c:\dev\python\hebban\datasets\src\datasets\utils\py_utils.py", line 330, in _single_map_nested
    return function(data_struct)
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 930, in _build_single_dataset
    ds = self._as_dataset(
  File "c:\dev\python\hebban\datasets\src\datasets\builder.py", line 1006, in _as_dataset
    return Dataset(fingerprint=fingerprint, **dataset_kwargs)
  File "c:\dev\python\hebban\datasets\src\datasets\arrow_dataset.py", line 661, in __init__
    info = info.copy() if info is not None else DatasetInfo()
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 286, in copy
    return self.__class__(**{k: copy.deepcopy(v) for k, v in self.__dict__.items()})
  File "<string>", line 20, in __init__
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 176, in __post_init__
    self.task_templates = [
  File "c:\dev\python\hebban\datasets\src\datasets\info.py", line 177, in <listcomp>
    template.align_with_features(self.features) for template in (self.task_templates)
  File "c:\dev\python\hebban\datasets\src\datasets\tasks\text_classification.py", line 22, in align_with_features
    raise ValueError(f"Column {self.label_column} is not a ClassLabel.")
ValueError: Column review_sentiment is not a ClassLabel.
```

## Environment info
- `datasets` version: 2.4.1.dev0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.8
- PyArrow version: 8.0.0
- Pandas version: 1.4.3


Added dataset information in clinic oos dataset card

This PR aims to add relevant information like the Description, Language and citation information of the clinic oos dataset card.


Easily create loading script for benchmark comprising multiple huggingface datasets

Hi ! I think the simplest is to copy paste the `_split_generators` code from the other datasets and do a bunch of if-else, as in the glue dataset: https://huggingface.co/datasets/glue/blob/main/glue.py#L467

Hi,

I would like to create a loading script for a benchmark comprising multiple huggingface datasets.
The function _split_generators needs to return the files for the respective dataset. However, the files are not always in the same location for each dataset. I want to just make a wrapper dataset that provides a single interface to all the underlying datasets.
I thought about downloading the files with the load_dataset function and then providing the link to the cached file. But this seems a bit inelegant to me. What approach would you propose to do this?

Please let me know if you have any questions.

Cheers,
Joel


Add image classification processing guide

This PR follows up on #4710 to separate the object detection and image classification guides. It expands a little more on the original guide to include a more complete example of loading and transforming a whole dataset.


Shard parquet in `download_and_prepare`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4747). All of your documentation changes will be reflected on that endpoint.

Following https://github.com/huggingface/datasets/pull/4724 (needs to be merged first)

It's good practice to shard parquet files to enable parallelism with spark/dask/etc.

I added the `max_shard_size` parameter to `download_and_prepare` (default to 500MB for parquet, and None for arrow).

```python
from datasets import *

cache_dir = "s3://..."
builder = load_dataset_builder("squad", cache_dir=cache_dir)
builder.download_and_prepare(file_format="parquet", max_shard_size="5MB")
```

### Implementation details

The examples are written to a parquet file until `ParquetWriter._num_bytes > max_shard_size`. When this happens, a new writer is instantiated to start writing the next shard. At the end, all the shards are renamed to include the total number of shards in their names: `{builder.name}-{split}-{shard_id:05d}-of-{num_shards:05d}.parquet`

TODO:
- [x] docstrings
- [x] docs
- [x] tests

cc @severo


Dataset Viewer issue for yanekyuk/wikikey

The dataset is empty, as far as I can tell: there are no files in the repository at https://huggingface.co/datasets/yanekyuk/wikikey/tree/main

Maybe the viewer can display a better message for empty datasets

### Link

_No response_

### Description

_No response_

### Owner

_No response_


Allow `list_datasets` to include private datasets

enhancement

New feature or request

Thanks for opening this issue :)

If it can help, I think you can already use `huggingface_hub` to achieve this:
```python
>>> from huggingface_hub import HfApi
>>> [ds_info.id for ds_info in HfApi().list_datasets(use_auth_token=token) if ds_info.private]
['bigscience/xxxx', 'bigscience-catalogue-data/xxxxxxx', ... ]
```

---------

Though the latest versions of `huggingface_hub` that contain this feature are not available on python 3.6, so maybe we should first drop support for python 3.6 (see #4460) to update `list_datasets` in `datasets` as well (or we would have to copy/paste some `huggingface_hub` code)

Great, thanks @lhoestq the workaround works! I think it would be intuitive to have the support directly in `datasets` but it makes sense to wait given that the workaround exists :)

i also think that going forward we should replace more and more implementations inside datasets with the corresponding ones from `huggingface_hub` (same as we're doing in `transformers`)

I am working with a large collection of private datasets, it would be convenient for me to be able to list them.

I would envision extending the convention of using `use_auth_token` keyword argument to `list_datasets` function, then calling:

```
list_datasets(use_auth_token="my_token")
```

would return the list of all datasets I have permissions to view, including private ones. The only current alternative I see is to use the hub website to manually obtain the list of dataset names - this is in the context of BigScience where respective private spaces contain hundreds of datasets, so not very convenient to list manually.