# Multiprocessing and sharding when creating dataset from scratch using loading script

**URL:** <https://discuss.huggingface.co/t/multiprocessing-and-sharding-when-creating-dataset-from-scratch-using-loading-script/25190>\
**Category:** 🤗Datasets\
**Created:** [October 29, 2022, 4:16pm UTC](https://discuss.huggingface.co/t/multiprocessing-and-sharding-when-creating-dataset-from-scratch-using-loading-script/25190 "2022-10-29T16:16:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![parano](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@parano](https://discuss.huggingface.co/u/parano)\
**Post date:** [October 29, 2022, 4:16pm UTC](https://discuss.huggingface.co/t/multiprocessing-and-sharding-when-creating-dataset-from-scratch-using-loading-script/25190/1 "2022-10-29T16:16:01Z")

</div>

Hi guys …

We are currently dealing with huge number of images which defintley wont fit the memory of our workstations, we wrote a couple of loading scripts following the tutorial from [here](https://huggingface.co/docs/datasets/dataset_script) and saw that it will take decades to generate the dataset using a single core … here we encounter with a couple of questions, first is that, does the `GeneratorBasedBuilder` class supports `multiprocessing` in a plug and play fashion (multiprocessing need not to be written by us) second since the dataset is huge we found out that while generating the dataset using `load_dataset` function it loads all the data to the `computer's memory` is there anyway that makes it to be persistent on the disk instead of memory?

BR

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [November 4, 2022, 3:02pm UTC](https://discuss.huggingface.co/t/multiprocessing-and-sharding-when-creating-dataset-from-scratch-using-loading-script/25190/2 "2022-11-04T15:02:56Z")

</div>

Hey ! We’re adding multiprocessing here: [Multiprocessed dataset builder [WIP] by TevenLeScao · Pull Request #5107 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/pull/5107)

In the next version of `datasets` you’ll be able to do

```python
num_proc = 16 # choose the number of processes to run in parallel
ds = load_dataset(..., num_proc=num_proc)

```

Each process is given a subset of shards to process - so just make sure your dataset is made of many shards 🙂

You can already install `datasets` from this branch if you want to try it out.

Re: memory

`datasets` flushes the data on disk every batch 10,000 items per process (in the next release it will be decreased to 1,000). So as long as the batches fit in memory you are fine. You can set a custom dataset writer batch size by setting the `DEFAULT_WRITER_BATCH_SIZE` class attribute of your dataset builder. It can be useful to set it to a lower value to reduce the memory usage.

---

<div class="post-metadata">

**Author:** ![parano](https://avatars.discourse-cdn.com/v4/letter/p/d2c977/32.png) [@parano](https://discuss.huggingface.co/u/parano)\
**Post date:** [November 4, 2022, 3:17pm UTC](https://discuss.huggingface.co/t/multiprocessing-and-sharding-when-creating-dataset-from-scratch-using-loading-script/25190/3 "2022-11-04T15:17:03Z")

</div>

Nice …

Let me know if I can contribute to any of the stuff …