Hello everyone,

I'm quite new to transformers. I started this competition to learn more about them. Thanks to hugging face's transformers library things are pretty simple to put in place.

I've tried BERT (base-uncased), Roberta (base and large), and Distilbert and so far the best score I got (with Roberta) is around 0.37...

The classifier I put on top of them is a simple linear layer with four 4 outputs. (multi-label classification)

I also noticed that these models easily overfit on the dataset (in a few epochs like 4 or 5).

What is your experience with transformers?

Do you have any tips/best practices to share, in particular on small datasets with short phrases?

Let's learn together :)