Add ModernBERT config

[jamt9000]

Adds a ModernBERT config for the original toxic comment classification challenge, using the ModernBERT-base model.

ModernBERT is an architecture similar to BERT leveraging more recent techniques like RoPE for long context and flash attention for faster inference. It is also trained on a variety of mainly English sources, including scientific articles and code.

Currently requires installing transformers from git to train: huggingface/transformers#35362 (comment)

CUDA_VISIBLE_DEVICES=1 python train.py -c configs/Toxic_comment_classification_ModernBERT.json

[jamt9000]

Currently the modernbert forward pass gives NaN outputs:

outputs
tensor([[    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [-0.8802, -0.9810, -0.5694, -0.4875,  0.7263,  0.0181],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan],
        [    nan,     nan,     nan,     nan,     nan,     nan]],
       device='cuda:0')

Edit: Seems to be fixed after installing flash attention pip install flash-attn --no-build-isolation

[jamt9000]

Might be worth using the hyperparameters given here:

https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb

    "optimizer": {
        "type": "AdamW",
        "args": {
            "lr": 8e-5,
            "weight_decay": 8e-6,
            "betas": [0.9, 0.98],
            "eps": 1e-6,
            "amsgrad": false
        }
    }

[laurahanu]

nice! would be good to have something about the benefits of modernbert in the description too

[m13uz]

Let's go!