# Same metrics after every epoch **URL:** \ **Category:** Beginners\ **Created:** [May 27, 2024, 10:22pm UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851 "2024-05-27T22:22:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![fmppo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fmppo/32/27241_2.png) [@fmppo](https://discuss.huggingface.co/u/fmppo)\ **Post date:** [May 27, 2024, 10:22pm UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851/1 "2024-05-27T22:22:37Z")
This is a duplicate of a question I’ve posted on Stack Overflow that until now has gotten no answers (link to [SO post](https://stackoverflow.com/questions/78492018/hugging-face-trainer-object-returning-constant-values-for-metrics-every-epoch)). Basically, I’m using the Trainer class to fine tune a model on a text classification task and the metrics I’m tracking are taking on the same values after every epoch - changing LR and batch size does not seem to change anything. This is the code I’m currently using: ```auto def compute_metrics(eval_preds): logits, labels = eval_preds preds = np.argmax(logits, axis=-1) acc = accuracy_score(labels, preds) recall = recall_score(labels, preds, average='macro') precision = precision_score(labels, preds, average='macro') f1 = f1_score(labels, preds, average='macro') metrics = { 'accuracy': acc, 'macro_f1': f1, 'macro_recall': recall, 'macro_precision': precision } return metrics def fine_tune_transformer(model_ckpt, dataset, output_dir, max_len=256, batch_size=16, epochs=3, learning_rate=1e-5, num_labels=2): device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = AutoModelForSequenceClassification.from_pretrained(model_ckpt, num_labels=num_labels).to(device) tokenizer = AutoTokenizer.from_pretrained(model_ckpt) def tokenize(batch): return tokenizer(batch['text'], padding='max_length', truncation=True, max_length=max_len) dataset_encoded = dataset.map(tokenize, batched=True, batch_size=None) os.makedirs(output_dir, exist_ok=True) training_args = TrainingArguments(output_dir=output_dir, num_train_epochs=epochs, learning_rate=learning_rate, per_device_train_batch_size=batch_size, per_device_eval_batch_size=batch_size, weight_decay=0.01, evaluation_strategy="epoch", disable_tqdm=False, push_to_hub=False, log_level="error") trainer = Trainer(model=model, args=training_args, compute_metrics=compute_metrics, train_dataset=dataset_encoded['train'], eval_dataset=dataset_encoded['validation'], tokenizer=tokenizer) return trainer model_ckpt = 'xlm-roberta-base' output_dir = 'content/drive/MyDrive/outputs' trainer = fine_tune_transformer(model_ckpt, joint_dataset, output_dir) trainer.train() ``` Please see here an example of the callback that shows the metrics after every epoch: ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/a/8/a870b08225c2e28700fe06e7530f8fdcb484224d.png) I’ve also checked the logits after every epoch and they are different, see example below: ```auto # epoch 1 # [[1.4335482 -1.6775645] # [-0.59078467 0.49235147] # [1.3932185 -1.6217263] # ... # [1.5345982 -1.7932017] # [1.6025882 -1.8546286] # [-0.59078467 0.49235126]] # epoch 2: # [[1.5743057 -1.806836] # [-0.5966272 0.5360965] # [1.5249034 -1.7561252] # ... # [1.594467 -1.8229653] # [1.5586028 -1.7579104] # [-0.596627 0.53609663]] ``` Please be noted that I’ve implemented the same code but fitting the model using Pytorch (not showing code due to its length, but let me know if needed and I’ll edit my response!) instead of the Trainer class and the training process is working as expected, hence I believe I’m doing something wrong with my use of the Trainer class. ---
**Author:** ![Zemulax](https://avatars.discourse-cdn.com/v4/letter/z/48db29/32.png) [@Zemulax](https://discuss.huggingface.co/u/Zemulax)\ **Post date:** [May 28, 2024, 12:59am UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851/2 "2024-05-28T00:59:27Z")
How is the inference? How much data are yu finetuning on? I have a feeling it might be because the model quickly reaches its optimal with those values. Also try a lr of 100 and see what happens ---
**Author:** ![fmppo](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/fmppo/32/27241_2.png) [@fmppo](https://discuss.huggingface.co/u/fmppo)\ **Post date:** [May 30, 2024, 12:32pm UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851/3 "2024-05-30T12:32:38Z")
Apologies for the late reply. You are probably right - the dataset I am using can be so straight forward at times that after digging deeper into it I was able to develop a rule-based model that got similar results. Thank you! ---
**Author:** ![Zemulax](https://avatars.discourse-cdn.com/v4/letter/z/48db29/32.png) [@Zemulax](https://discuss.huggingface.co/u/Zemulax)\ **Post date:** [May 30, 2024, 12:48pm UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851/4 "2024-05-30T12:48:30Z")
No problem. Well done! ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [May 31, 2024, 12:49am UTC](https://discuss.huggingface.co/t/same-metrics-after-every-epoch/88851/5 "2024-05-31T00:49:18Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.