# Datasets with custom python objects

**URL:** <https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050>\
**Category:** 🤗Datasets\
**Created:** [March 28, 2024, 3:20am UTC](https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050 "2024-03-28T03:20:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fzyzcjy/32/25229_2.png) [@fzyzcjy](https://discuss.huggingface.co/u/fzyzcjy)\
**Post date:** [March 28, 2024, 3:20am UTC](https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050/1 "2024-03-28T03:20:38Z")

</div>

Hi thanks for the library! I would like to have a huggingface Dataset, and one of its column is custom (non-serializable) Python objects. For example, a minimal code:

```auto
class MyClass:
    pass

dataset = datasets.Dataset.from_list([
    dict(a=MyClass(), b='hello'),
])

```

It gives error:

```auto
ArrowInvalid: Could not convert < __main__.MyClass object at 0x7a852830d050> with type MyClass: did not recognize Python value type when inferring an Arrow data type

```

I guess it is because Dataset forces to convert everything into arrow format. However, is there any ways to make the scenario work? Thanks!

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [April 3, 2024, 12:48pm UTC](https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050/2 "2024-04-03T12:48:37Z")

</div>

Hi ! Indeed Arrow doesn’t support serializing arbitrary objects and only supports types like integers, floats, strings, lists, dicts, etc.

You could try converting your objects to JSON (string) or to Pickle (bytes), and then rebuild your object from this when needed

---

<div class="post-metadata">

**Author:** ![fzyzcjy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fzyzcjy/32/25229_2.png) [@fzyzcjy](https://discuss.huggingface.co/u/fzyzcjy)\
**Post date:** [April 3, 2024, 2:35pm UTC](https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050/3 "2024-04-03T14:35:51Z")

</div>

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [April 4, 2024, 2:36am UTC](https://discuss.huggingface.co/t/datasets-with-custom-python-objects/79050/4 "2024-04-04T02:36:45Z")

</div>

Closed on April 4, 2024, 2:36am UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.