ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'

[exs-avianello]

Describe the bug

Saving a dataset .to_json() fails with a ValueError since the latest pandas release (2.1.0)

In their latest release we have:

Improved error handling when using DataFrame.to_json() with incompatible index and orient arguments (GH 52143)

i.e. an error is now raised for invalid combinations of index and orient.

This means that unfortunately the custom logic at this line might sometimes lead to contradictions:

datasets/src/datasets/io/json.py

Line 96 in 029227a

index = self.to_json_kwargs.pop("index", False if orient in ["split", "table"] else True)

e.g. for the default case orient=records leads to index=True, which now raises a ValueError

Steps to reproduce the bug

import datasets


if __name__ == '__main__':

    dataset = datasets.Dataset.from_dict({"A": [1, 2, 3], "B": [4, 5, 6]})
    dataset.to_json("dataset.json")

>>>
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'.

Expected behavior

The dataset is successfully saved as .json

Environment info

python >= 3.9
pandas >= 2.1.0

[albertvillanova]

Thanks for reporting. We are investigating it.

[albertvillanova]

This issue is caused by latest pandas release 2.1.0 (released yesterday Aug 30).

See: https://github.com/huggingface/datasets/actions/runs/6035484010/job/16375932085?pr=6198

[albertvillanova]

People using previous releases of datasets should pin pandas in their local environment:

python -m pip install 'pandas<2.1.0'