# Datasets not behaving as expected after random data augmentation with map

**URL:** <https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100>\
**Category:** 🤗Datasets\
**Created:** [September 18, 2021, 4:02pm UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100 "2021-09-18T16:02:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![SMMousavi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smmousavi/32/2314_2.png) [@SMMousavi](https://discuss.huggingface.co/u/SMMousavi)\
**Post date:** [September 18, 2021, 4:02pm UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/1 "2021-09-18T16:02:12Z")

</div>

I’m using [nlpaug](https://github.com/makcedward/nlpaug) to augment a split of the sst2 dataset. As instructed in the [documentation](https://huggingface.co/docs/datasets/process.html#batch-processing), I’m using `map` with `batched=True` for this purpose. The function I pass to `map` takes one instance (`batch_size=1`) and generates several instances. The important thing here is that this function is not a pure function, the sentence it generates and the number of instances it generates and returns are completely random. I get a warning each time that says there is a problem with caching and fingerprint, which I assumed is because of the random nature of my function.

After the data augmentation, the dataset acts weird; for example:

```auto
synonym_aug_datasets[0.2]["train"].filter(lambda x: x['idx'] > 200000)["idx"][:10]

```

synonym\_aug\_datasets is a python dictionary with several augmented datasets in it. I’m simply filtering all instances with `idx` larger than 200000 and then looking at the `idx` of the first 10 instances, all of them should have `idx` larger than 200000, right? **Every time I run this code, I get a different result, sometimes even with `idx` smaller than 200000, and sometimes it won’t even run**  
output 1:

```auto
[55154, 55917, 200628, 409, 6218, 33825, 201639, 2063, 49115, 2959]

```

output 2:

```auto
ValueError: Keys mismatch: between {'indices': Value(dtype='uint64', id=None)} and {'idx': Value(dtype='int32', id=None), 'label': Value(dtype='int64', id=None), 'sentence': Value(dtype='string', id=None)}

```

output 3:

```auto
[200205,
 200168,
 200888,
 200157,
 201597,
 201784,
 200899,
 200466,
 201086,
 200435]

```

How is that even possible?

---

<div class="post-metadata">

**Author:** ![adorkin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adorkin/32/2704_2.png) [@adorkin](https://discuss.huggingface.co/u/adorkin)\
**Post date:** [September 20, 2021, 11:05am UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/2 "2021-09-20T11:05:41Z")

</div>

Well, can’t you show the function that you’re passing to map? It’s pretty difficult to figure out what goes wrong otherwise.

> [@SMMousavi](#):
>
> the sentence it generates and the number of instances it generates and returns are completely random

I guess that the return value of that function might be the culprit. The logic behind map is that it passes the current batch (which is basically a dict-like object) to the function, takes its output (which should be a dictionary with keys being feature names) and adds or replaces (if names match) key value pairs in the given batch object. So the question is how do you implement the case where multiple augmented sentences are returned by the function? Does your `sentence` feature become a sequence of strings? Or do you do something else?

So to reiterate, the way you’re returning multiple augmented sentences for a single input sentence might be the cause.

---

<div class="post-metadata">

**Author:** ![SMMousavi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smmousavi/32/2314_2.png) [@SMMousavi](https://discuss.huggingface.co/u/SMMousavi)\
**Post date:** [September 20, 2021, 9:04pm UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/3 "2021-09-20T21:04:52Z")

</div>

> [@adorkin](#):
>
> So the question is how do you implement the case where multiple augmented sentences are returned by the function? Does your `sentence` feature become a sequence of strings? Or do you do something else?

In the documentation, I read that I can use map with `batched=True` to change the size of a dataset, returning fewer or more instances; unfortunately, I can not find it now in the documentation.

> [@adorkin](#):
>
> Well, can’t you show the function that you’re passing to map? It’s pretty difficult to figure out what goes wrong otherwise.

It’s a really complicated function, that’s why I didn’t post it, but here it is:

```auto
def _aug_map(self, example, aug_function, idx_generator):
        if example["label"][0] != self.minority_class:
            return example
        self.unique_aug = prob_round(self.num_each_sentence_aug)
        if self.unique_aug == 0:
            return example
        sentence = example["sentence"][0]
        augmented_list = pydash.flatten_deep([aug_function.augment(sentence, n=self.unique_aug)])
        label_list = [self.minority_class for _ in range(len(augmented_list))]
        
        idx_list = [next(idx_generator) + i for i in range(len(augmented_list))]

        example["sentence"].extend(augmented_list)
        example["label"].extend(label_list)
        example["idx"].extend(idx_list)
        return example

```

With lambda, I create another function that takes only `example` and calls this function with other required arguments, and then I pass that function to `map`.

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 21, 2021, 10:29am UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/4 "2021-09-21T10:29:42Z")

</div>

Hi !  
First, the output 2 that you got `ValueError: Keys mismatch` comes from a cache issue that has been fixed on the `master` branch of `datasets` recently, see [Backwards compatibility broken for cached datasets that use `.filter()` · Issue #2943 · huggingface/datasets · GitHub](https://github.com/huggingface/datasets/issues/2943)

Then, regarding the other output, can you check the logs if it reloads previously computed results from the cache or not ?

---

<div class="post-metadata">

**Author:** ![SMMousavi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smmousavi/32/2314_2.png) [@SMMousavi](https://discuss.huggingface.co/u/SMMousavi)\
**Post date:** [September 21, 2021, 11:42am UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/5 "2021-09-21T11:42:56Z")

</div>

Let me tell you my real issue. With the augment function that I shared in my previous post, I’m creating different versions of augmented datasets, but I get the exact same result each time I train my model on them.  
Datasets library is acting really, really weird. Before I train my model on several augmented datasets, I check and each dataset is different since it was augmented with different arguments and methods, but after training, when I check them, they are exactly the same.

When I train my model on them, in the log, each time before training starts, the datasets library is loading the exact same cache while the datasets are not actually the same.

Generating different versions of augmented datasets:

```python
synonym_aug_datasets = {}

for aug_p in np.linspace(0.1, 0.5, 5):
    balanced_train = dataset_aug.synonym_augment(aug_p=aug_p)
    balanced_datasets = DatasetDict({
        "train": balanced_train.shuffle(SEED).flatten_indices(),
        "validation": imbalanced_datasets["validation"].flatten_indices(),
        "test": imbalanced_datasets["test"].flatten_indices(),
    })
    synonym_aug_datasets[aug_p] = balanced_datasets

```

Training on these augmented datasets:

```python
synonym_aug_result = {}

for aug_p, balanced_datasets in synonym_aug_datasets.items():
    preds = custom_train(balanced_datasets)
    synonym_aug_result[aug_p] = preds

```

`synonym_aug_result` contains the same preds for each `aug_p`.

1st time:  
Not loading from the cache, processing

2nd time loading from the cache:

```nohighlight
Loading cached processed dataset at /tmp/tmpwwka_321/cache-5a9967131defac9d.arrow
Loading cached processed dataset at /tmp/tmpwwka_321/cache-b9a54799246ef5f7.arrow
Loading cached processed dataset at /tmp/tmpwwka_321/cache-7f6e5f5590fb6dfa.arrow

```

3rd time loading from the cache:

```nohighlight
Loading cached processed dataset at /tmp/tmpwwka_321/cache-5a9967131defac9d.arrow
Loading cached processed dataset at /tmp/tmpwwka_321/cache-b9a54799246ef5f7.arrow
Loading cached processed dataset at /tmp/tmpwwka_321/cache-7f6e5f5590fb6dfa.arrow

```

My `custom_train` function is basically written from the sample on the tutorial course:

```python
def custom_train(balanced_datasets, checkpoint=CHECKPOINT, seed=SEED, saving_folder=SAVING_FOLDER):
    set_seed(seed)
    model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)
    tokenizer = AutoTokenizer.from_pretrained(checkpoint)
    
    def tokenize_function(example):
        return tokenizer(example["sentence"], truncation=True)

    tokenized_datasets = balanced_datasets.map(tokenize_function, batched=True)
    data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

    saving_folder = SAVING_FOLDER + "_balanced"
    training_args = TrainingArguments(
        saving_folder,
    )

    trainer = Trainer(
        model,
        training_args,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["validation"],
        data_collator=data_collator,
        tokenizer=tokenizer,
    )
    
    trainer.train()
    
    predictions = trainer.predict(tokenized_datasets["test"])
    preds = np.argmax(predictions.predictions, axis=-1)
    
    return preds

```

---

<div class="post-metadata">

**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/64813_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\
**Post date:** [September 21, 2021, 2:43pm UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/6 "2021-09-21T14:43:51Z")

</div>

If I understand correctly, you are showing the logs of the training, that you did three times on different datasets (first run aug\_p=0.1, second run aug\_p=0.2, third run aug\_p=0.3) ?  
The 2nd and 3rd times it’s loading from the cache the tokenized data. It means that it considers that the dataset you passed (even though aug\_p changed) is the same as before.

An easy way to debug this and see if this is the case would be to do

```python
for aug_p, balanced_datasets in synonym_aug_datasets.items():
    print(balanced_datasets["train"]._fingerprint)

```

If the fingerprints are the same, then it means the datasets are the same.  
Could you verify the fingerprints please ? This should be helpful to debug things.

However your function is random, and you said that you got a warning about the cache - it must have notified you that it used random fingerprints because of the complex nature of your function.

Can you confirm this is what the warning message said ?

---

<div class="post-metadata">

**Author:** ![SMMousavi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smmousavi/32/2314_2.png) [@SMMousavi](https://discuss.huggingface.co/u/SMMousavi)\
**Post date:** [September 22, 2021, 12:31am UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/7 "2021-09-22T00:31:29Z")

</div>

> [@lhoestq](#):
>
> If I understand correctly, you are showing the logs of the training, that you did three times on different datasets (first run aug\_p=0.1, second run aug\_p=0.2, third run aug\_p=0.3) ?  
> The 2nd and 3rd times it’s loading from the cache the tokenized data. It means that it considers that the dataset you passed (even though aug\_p changed) is the same as before.

Exactly.

> [@lhoestq](#):
>
> If the fingerprints are the same, then it means the datasets are the same.  
> Could you verify the fingerprints please ? This should be helpful to debug things.

Of course. Here it is:

```plaintext
cec3c64992040ae1
dced09bea689051f
7711c97e1932d34d
d86609c29dc217a0
f1b80214a0a91cfc

```

But after training, I get the exact same result. It loads the exact same cache each time, just as I told you in my previous post.

> [@lhoestq](#):
>
> However your function is random, and you said that you got a warning about the cache - it must have notified you that it used random fingerprints because of the complex nature of your function.
> 
> Can you confirm this is what the warning message said ?

I saw this warning before, and yes, it said it can’t hash my function, but now I don’t see this. There is no warning when I augment my dataset now.

---

<div class="post-metadata">

**Author:** ![SMMousavi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/smmousavi/32/2314_2.png) [@SMMousavi](https://discuss.huggingface.co/u/SMMousavi)\
**Post date:** [September 23, 2021, 12:32am UTC](https://discuss.huggingface.co/t/datasets-not-behaving-as-expected-after-random-data-augmentation-with-map/10100/8 "2021-09-23T00:32:09Z")

</div>

I found a solution for now. Just before tokenizing, I’m converting my datasets to pandas data frame and converting them back to datasets. By doing this, the datasets library doesn’t load the same cache each time and recognizes that my datasets are different.

```python
new_datasets = DatasetDict({
    "train": Dataset.from_pandas(old_datasets["train"].to_pandas()),
    "validation": Dataset.from_pandas(old_datasets["validation"].to_pandas()),
    "test": Dataset.from_pandas(old_datasets["test"].to_pandas()),
})

```