Issues Fine Tuning for new Task

[arvisioncode]

Hi,

I am fine-tuning a new key-value extraction task for the Florence model. I started from the notebook from https://colab.research.google.com/drive/1hKDrJ5AH_o7I95PtZ9__VlCTNAo1Gjpf?usp=sharing#scrollTo=zqDWEWDcaSxN

Parameters:

EPOCHS = 200
LR = 2e-6
Model base = microsoft/Florence-2-base-ft

The problem I encountered during training was that the learning capacity for this task was deficient. The model manages to learn the first few key-value pairs but never learns the rest, only the first ones. As a result, the length of the predictions is much shorter than that of the ground truth.

Why is it that the model does not learn all the content of the ground truth?
Is there a learning limitation?
Is there any way to solve this problem?

Regards