Add support for arrow iterable when concatenating or interleaving

[radulescupetru]

Fixes a case when concatenating or interleaving datasets with with_format(...) call was slower.
Details here: #6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.

[radulescupetru]

Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])

[lhoestq]

Cool ! Love the speed up :)

Note that __iter__ is used to iterate on python objects though, not pyarrow tables.
Would it be possible to take that into account and implement proper _iter_arrow methods when necessary instead of using __iter__, as in your first proposition in the original issue ?

You can probably still minimize the duplication between iter and iter_arrow methods by making __iter__ and _iter_arrow share some code.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[radulescupetru]

@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci?

[lhoestq]

awesome ! lgtm :)

[radulescupetru]

@lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?

[lhoestq]

They seem unrelated to your changes. Merging :)