# Iterable datasets for array data, limited formatting options

**URL:** <https://discuss.huggingface.co/t/iterable-datasets-for-array-data-limited-formatting-options/62677>\
**Category:** 🤗Datasets\
**Created:** [November 18, 2023, 2:57pm UTC](https://discuss.huggingface.co/t/iterable-datasets-for-array-data-limited-formatting-options/62677 "2023-11-18T14:57:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rmko4](https://avatars.discourse-cdn.com/v4/letter/r/dc4da7/32.png) [@Rmko4](https://discuss.huggingface.co/u/Rmko4)\
**Post date:** [November 18, 2023, 2:57pm UTC](https://discuss.huggingface.co/t/iterable-datasets-for-array-data-limited-formatting-options/62677/1 "2023-11-18T14:57:14Z")

</div>

Iterable datasets have limited formatting options, which can be problematic/awkward when using array data. Right now, when applying a function to the dataset with ‘map’, the default Python formatter is used. This will convert an array to a Python list. This is incredibly slow when dealing with large (multi-dimensional) arrays.

It would be convenient if other formatters are supported, such as Numpy. The Arrow formatter is (partially) supported but in certain cases it appears to still use a python format for certain iterators, making it very slow. Furthermore, this requires more advanced knowledge of Apache Arrow.

The workaround that I applied, is to encode the arrays to a binary format. When loading the dataset as an iterable dataset the binary format is then first ‘decoded’ to a Numpy array prior to applying the transform. A torch format is applied to receive the processed data in the desired format.

— My questions:  
**Is there a better way to work with array data when using the iterable dataset?** I am not using Image data, so using the image feature will not be of much use.  
**Are there plans to support other formatting options in the future?**

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 29, 2023, 5:08pm UTC](https://discuss.huggingface.co/t/iterable-datasets-for-array-data-limited-formatting-options/62677/2 "2023-11-29T17:08:12Z")

</div>

You can use the NumPy formatting

```python
ds = ds.with_format("numpy")

```

then you can iterate on it or `map` and get numpy arrays

```python
for example_with_numpy_arrays in ds:
    ...

```

for map:

```python
ds = ds.map(my_func_that_uses_numpy_arrays)

```

---

<div class="post-metadata">

**Author:** ![Rmko4](https://avatars.discourse-cdn.com/v4/letter/r/dc4da7/32.png) [@Rmko4](https://discuss.huggingface.co/u/Rmko4)\
**Post date:** [December 28, 2023, 7:55pm UTC](https://discuss.huggingface.co/t/iterable-datasets-for-array-data-limited-formatting-options/62677/3 "2023-12-28T19:55:40Z")

</div>

@lhoestq thank you for your response! I have a working solution, but I believe I couldn’t quite get it to work with the method you suggested. Perhaps I have some time to look into it later.