Update dependency datasets to v2.16.1 - autoclosed

[renovate]

This PR contains the following updates:

Package	Change	Age	Adoption	Passing	Confidence
datasets	2.15.0 -> 2.16.1

---

Release Notes

huggingface/datasets (datasets)

v2.16.1

Compare Source

Bug fixes

• Fix dl_manager.extract returning FileNotFoundError by @​lhoestq in https://github.com/huggingface/datasets/pull/6543
  • Fix bug causing FileNotFoundError when passing a relative directory as cache_dir to load_dataset
• Fix custom configs from script by @​lhoestq in https://github.com/huggingface/datasets/pull/6544
  • Fix bug when loading a dataset with a loading script using custom arguments would fail
  • e.g. load_dataset("ted_talks_iwslt", language_pair=("ja", "en"), year="2015")

Full Changelog: huggingface/datasets@2.16.0...2.16.1

v2.16.0

Compare Source

Security features

• Add trust_remote_code argument by @​lhoestq in https://github.com/huggingface/datasets/pull/6429
  • Some Hugging Face datasets contain custom code which must be executed to correctly load the dataset. The code can be inspected in the repository content at https://hf.co/datasets/<repo_id>. A warning is shown to let the user know about the custom code, and they can avoid this message in future by passing the argument trust_remote_code=True.
  • Passing trust_remote_code=True will be mandatory to load these datasets from the next major release of datasets.
  • Using the environment variable HF_DATASETS_TRUST_REMOTE_CODE=0 you can already disable custom code by default without waiting for the next release of datasets
• Use parquet export if possible by @​lhoestq in https://github.com/huggingface/datasets/pull/6448
  • This allows loading most old datasets based on custom code by downloading the Parquet export provided by Hugging Face
  • You can see a dataset's Parquet export at https://hf.co/datasets/<repo_id>/tree/refs%2Fconvert%2Fparquet

Features

• Webdataset dataset builder by @​lhoestq in https://github.com/huggingface/datasets/pull/6391
• Implement get dataset default config name by @​albertvillanova in https://github.com/huggingface/datasets/pull/6511
• Lazy data files resolution and offline cache reload by @​lhoestq in https://github.com/huggingface/datasets/pull/6493
  • This speeds up the load_dataset step that lists the data files of big repositories (up to x100) but requires huggingface_hub 0.20 or newer
  • Fix load_dataset that used to reload data from cache even if the dataset was updated on Hugging Face
  • Reload a dataset from your cache even if you don't have internet connection
  • New cache directory scheme for no-script datasets: ~/.cache/huggingface/datasets/username___dataset_name/config_name/version/commit_sha
  • Backward comaptibility: cached datasets from datasets 2.15 (using the old scheme) are still reloaded from cache

General improvements and bug fixes

• Remove unused argument in _get_data_files_patterns by @​lhoestq in https://github.com/huggingface/datasets/pull/6343
• Set usedforsecurity=False in hashlib methods (FIPS compliance) by @​Wauplin in https://github.com/huggingface/datasets/pull/6414
• Use ruff for formatting by @​mariosasko in https://github.com/huggingface/datasets/pull/6434
• Create DatasetNotFoundError and DataFilesNotFoundError by @​albertvillanova in https://github.com/huggingface/datasets/pull/6431
• Fix multi gpu map example by @​lhoestq in https://github.com/huggingface/datasets/pull/6415
• Better tqdm wrapper by @​mariosasko in https://github.com/huggingface/datasets/pull/6433
• Remove Table.__getstate__ and Table.__setstate__ by @​LZHgrla in https://github.com/huggingface/datasets/pull/6444
• Use filelock package for file locking by @​mariosasko in https://github.com/huggingface/datasets/pull/6445
• Fix metadata file resolution when inferred pattern is ** by @​mariosasko in https://github.com/huggingface/datasets/pull/6449
• Update hub-docs reference by @​mishig25 in https://github.com/huggingface/datasets/pull/6453
• Refactor dill logic by @​mariosasko in https://github.com/huggingface/datasets/pull/6454
• Don't require trust_remote_code in inspect_dataset by @​lhoestq in https://github.com/huggingface/datasets/pull/6456
• [docs] troubleshooting guide by @​MKhalusova in https://github.com/huggingface/datasets/pull/6424
• Missing DatasetNotFoundError by @​lhoestq in https://github.com/huggingface/datasets/pull/6462
• Disable benchmarks in PRs by @​lhoestq in https://github.com/huggingface/datasets/pull/6463
• More robust temporary directory deletion by @​mariosasko in https://github.com/huggingface/datasets/pull/6426
• Fix shard retry mechanism in push_to_hub by @​mariosasko in https://github.com/huggingface/datasets/pull/6461
• Use auth to get parquet export by @​lhoestq in https://github.com/huggingface/datasets/pull/6468
• Remove delete doc CI by @​lhoestq in https://github.com/huggingface/datasets/pull/6471
• Fix CI quality by @​albertvillanova in https://github.com/huggingface/datasets/pull/6473
• Fix PermissionError on Windows CI by @​albertvillanova in https://github.com/huggingface/datasets/pull/6477
• More robust preupload retry mechanism by @​mariosasko in https://github.com/huggingface/datasets/pull/6479
• Add IterableDataset __repr__ by @​lhoestq in https://github.com/huggingface/datasets/pull/6480
• Fix max lock length on unix by @​lhoestq in https://github.com/huggingface/datasets/pull/6482
• Fix ArrayXD YAML conversion by @​mariosasko in https://github.com/huggingface/datasets/pull/6168
• Fix docs phrasing about supported formats when sharing a dataset by @​albertvillanova in https://github.com/huggingface/datasets/pull/6486
• Fix deprecation warning when building conda package by @​albertvillanova in https://github.com/huggingface/datasets/pull/6425
• Make push_to_hub return CommitInfo by @​albertvillanova in https://github.com/huggingface/datasets/pull/6492
• docs: add reference Git over SSH by @​severo in https://github.com/huggingface/datasets/pull/6499
• Fallback on dataset script if user wants to load default config by @​lhoestq in https://github.com/huggingface/datasets/pull/6498
• Don't expand_info in HF glob by @​lhoestq in https://github.com/huggingface/datasets/pull/6469
• Fix streaming xnli by @​lhoestq in https://github.com/huggingface/datasets/pull/6503
• Pickle support for torch.Generator objects by @​mariosasko in https://github.com/huggingface/datasets/pull/6502
• Enable setting config as default when push_to_hub by @​albertvillanova in https://github.com/huggingface/datasets/pull/6500
• Better cast error when generating dataset by @​lhoestq in https://github.com/huggingface/datasets/pull/6509
• Replace list_files_info with list_repo_tree in push_to_hub by @​mariosasko in https://github.com/huggingface/datasets/pull/6510
• Remove deprecated HfFolder by @​lhoestq in https://github.com/huggingface/datasets/pull/6512
• Support huggingface-hub pre-releases by @​albertvillanova in https://github.com/huggingface/datasets/pull/6516
• Support push_to_hub canonical datasets by @​albertvillanova in https://github.com/huggingface/datasets/pull/6519
• Support commit_description parameter in push_to_hub by @​albertvillanova in https://github.com/huggingface/datasets/pull/6520
• fix get_metadata_patterns function args error by @​d710055071 in https://github.com/huggingface/datasets/pull/6518
• Fix metrics dead link by @​qgallouedec in https://github.com/huggingface/datasets/pull/6491
• fix tests by @​lhoestq in https://github.com/huggingface/datasets/pull/6523
• Cache backward compatibility with 2.15.0 by @​lhoestq in https://github.com/huggingface/datasets/pull/6514
• Preserve order of configs and splits when using Parquet exports by @​albertvillanova in https://github.com/huggingface/datasets/pull/6526

New Contributors

• @​LZHgrla made their first contribution in https://github.com/huggingface/datasets/pull/6444
• @​d710055071 made their first contribution in https://github.com/huggingface/datasets/pull/6518

Full Changelog: huggingface/datasets@2.15.0...2.16.0

---

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

---

• If you want to rebase/retry this PR, check this box

---

This PR has been generated by Mend Renovate. View repository job log here.