Code

Dev neil branch creation#23

Code

Open

Neilus03 wants to merge 181 commits intohitz-zentroa:mainhitz-zentroa/GoLLIE:mainfrom ikergarcia1996:dev-neilikergarcia1996/GoLLIE:dev-neilCopy head branch name to clipboard

+26,065-2,901

Commits

Commits on Oct 20, 2023

• Use HF flash attention and test new hparams

  ikergarcia1996

  committed
  8051dc3

  Copy full SHA for 8051dc3

• Merge pull request #1 from hitz-zentroa/main

  Show description for dc8d8c1
  ikergarcia1996

  authored
  dc8d8c1

  Copy full SHA for dc8d8c1

• Change name

  ikergarcia1996

  committed
  6a857c3

  Copy full SHA for 6a857c3

• Merge remote-tracking branch 'origin/dev' into dev

  ikergarcia1996

  committed
  4ce6f63

  Copy full SHA for 4ce6f63

• Set correct environment

  ikergarcia1996

  committed
  1590daa

  Copy full SHA for 1590daa

Commits on Oct 21, 2023

• Change hparams

  ikergarcia1996

  committed
  cf99869

  Copy full SHA for cf99869

• 2 GPUs

  ikergarcia1996

  committed
  5a10fbe

  Copy full SHA for 5a10fbe

• change project

  ikergarcia1996

  committed
  38d51c1

  Copy full SHA for 38d51c1

• Fix bug

  ikergarcia1996

  committed
  d2df066

  Copy full SHA for d2df066

• 128

  ikergarcia1996

  committed
  2a462ee

  Copy full SHA for 2a462ee

• Fix name

  ikergarcia1996

  committed
  42e95a5

  Copy full SHA for 42e95a5

• fix memory fragmentation

  ikergarcia1996

  committed
  c9f08f4

  Copy full SHA for c9f08f4

Commits on Oct 22, 2023

• Reduce batch size

  ikergarcia1996

  committed
  ac41590

  Copy full SHA for ac41590

• Use 4 GPUs

  ikergarcia1996

  committed
  d736865

  Copy full SHA for d736865

• Increase num threads

  ikergarcia1996

  committed
  9ab89a8

  Copy full SHA for 9ab89a8

• Evaluate all checkpoints

  ikergarcia1996

  committed
  4574408

  Copy full SHA for 4574408

Commits on Oct 23, 2023

• Attempt to fix dtype bug

  ikergarcia1996

  committed
  3495f41

  Copy full SHA for 3495f41

• Fix eval scripts

  ikergarcia1996

  committed
  4dabec4

  Copy full SHA for 4dabec4

• Fix names

  ikergarcia1996

  committed
  f579417

  Copy full SHA for f579417

• Marge model before inference

  ikergarcia1996

  committed
  9c813b2

  Copy full SHA for 9c813b2

• Merge models before inference

  ikergarcia1996

  committed
  a309f25

  Copy full SHA for a309f25

• Fix bug

  ikergarcia1996

  committed
  8ffa1c5

  Copy full SHA for 8ffa1c5

• bf16 eval

  ikergarcia1996

  committed
  d809598

  Copy full SHA for d809598

• Disable flash attention

  ikergarcia1996

  committed
  7d58fa7

  Copy full SHA for 7d58fa7

• Prepare run 128

  ikergarcia1996

  committed
  4c3e290

  Copy full SHA for 4c3e290

• Set correct output path

  ikergarcia1996

  committed
  d6d4b52

  Copy full SHA for d6d4b52

• Eval 256

  ikergarcia1996

  committed
  23d4fd0

  Copy full SHA for 23d4fd0

• Fix path

  ikergarcia1996

  committed
  eba9e1f

  Copy full SHA for eba9e1f

• Use flash attention

  ikergarcia1996

  committed
  ec4536a

  Copy full SHA for ec4536a

• Eval 128

  ikergarcia1996

  committed
  8cd3732

  Copy full SHA for 8cd3732

• Reduce RAM to 100GB

  ikergarcia1996

  committed
  ce090be

  Copy full SHA for ce090be

Commits on Oct 24, 2023

• hparam search

  ikergarcia1996

  committed
  2626ccb

  Copy full SHA for 2626ccb

• Fix data path

  ikergarcia1996

  committed
  33e5455

  Copy full SHA for 33e5455

• Add experiment

  ikergarcia1996

  committed
  f616c06

  Copy full SHA for f616c06

Commits on Oct 26, 2023

• Eval FT

  ikergarcia1996

  committed
  82c02bf

  Copy full SHA for 82c02bf

• eval

  ikergarcia1996

  committed
  670ea93

  Copy full SHA for 670ea93

• Remove ontonotes from eval (too slow for hparam search)

  ikergarcia1996

  committed
  beef851

  Copy full SHA for beef851

• Re run

  ikergarcia1996

  committed
  775bfe2

  Copy full SHA for 775bfe2

Commits on Oct 27, 2023

• Attempt to fix crash

  ikergarcia1996

  committed
  f316017

  Copy full SHA for f316017

• Do not run large datasets

  ikergarcia1996

  committed
  570a5d1

  Copy full SHA for 570a5d1

• Change paths

  ikergarcia1996

  committed
  8e0b0a7

  Copy full SHA for 8e0b0a7

• Change params

  ikergarcia1996

  committed
  ab69089

  Copy full SHA for ab69089

Commits on Oct 28, 2023

• Remove split size

  ikergarcia1996

  committed
  3f85820

  Copy full SHA for 3f85820

Commits on Nov 21, 2023

• Implement do not shuffle and disable paraphrase

  ikergarcia1996

  committed
  8078269

  Copy full SHA for 8078269

• Script to get seen and unseen labels

  ikergarcia1996

  committed
  6e942be

  Copy full SHA for 6e942be

Commits on Nov 22, 2023

• Implement loss from "text" and "all"

  ikergarcia1996

  committed
  46fcf5f

  Copy full SHA for 46fcf5f

• Remove environment

  ikergarcia1996

  committed
  26cb61d

  Copy full SHA for 26cb61d

• Fix typo

  ikergarcia1996

  committed
  19ab3d0

  Copy full SHA for 19ab3d0

• Install flash attention in the node

  ikergarcia1996

  committed
  099cd60

  Copy full SHA for 099cd60

Commits on Nov 28, 2023

• Fix model

  ikergarcia1996

  committed
  d4818e5

  Copy full SHA for d4818e5

• Set xlong partition

  ikergarcia1996

  committed
  ee2d42b

  Copy full SHA for ee2d42b

• Set correct partition

  ikergarcia1996

  committed
  f6f4c77

  Copy full SHA for f6f4c77

• Fix output path

  ikergarcia1996

  committed
  22fe23c

  Copy full SHA for 22fe23c

• Fix wandb

  ikergarcia1996

  committed
  268b2c4

  Copy full SHA for 268b2c4

Commits on Nov 29, 2023

• Change batch size

  ikergarcia1996

  committed
  11128b1

  Copy full SHA for 11128b1

Commits on Nov 30, 2023

• Reduce batch size

  ikergarcia1996

  committed
  f7e80f8

  Copy full SHA for f7e80f8

Commits on Dec 1, 2023

• DeepSeek

  ikergarcia1996

  committed
  678005c

  Copy full SHA for 678005c

• Deepseek

  ikergarcia1996

  committed
  ee1d7b6

  Copy full SHA for ee1d7b6

• Restore sh

  ikergarcia1996

  committed
  944d8ab

  Copy full SHA for 944d8ab

• Fix run command

  ikergarcia1996

  committed
  d0b7687

  Copy full SHA for d0b7687

Commits on Dec 8, 2023

• Fix eval

  ikergarcia1996

  committed
  95339e3

  Copy full SHA for 95339e3

• Deepseek

  ikergarcia1996

  committed
  46bac0d

  Copy full SHA for 46bac0d

Commits on Dec 9, 2023

• Increase GPU mem

  ikergarcia1996

  committed
  070b22d

  Copy full SHA for 070b22d

• Deepspeed

  ikergarcia1996

  committed
  4137bf9

  Copy full SHA for 4137bf9

Commits on Dec 10, 2023

• Fix eps string deepspeed

  ikergarcia1996

  committed
  9953ea5

  Copy full SHA for 9953ea5

• Eval

  ikergarcia1996

  committed
  561fc46

  Copy full SHA for 561fc46

• Test new models

  ikergarcia1996

  committed
  6396526

  Copy full SHA for 6396526

• Ensure 4 bit inference

  ikergarcia1996

  committed
  0d8b20c

  Copy full SHA for 0d8b20c

• Reduce hw

  ikergarcia1996

  committed
  299733c

  Copy full SHA for 299733c

• Fix

  ikergarcia1996

  committed
  34ffd7f

  Copy full SHA for 34ffd7f

• 8 gpus

  ikergarcia1996

  committed
  dc7e570

  Copy full SHA for dc7e570

Commits on Dec 11, 2023

• Fix optimizer

  ikergarcia1996

  committed
  561bccc

  Copy full SHA for 561bccc

Commits on Dec 12, 2023

• Tune params

  ikergarcia1996

  committed
  a82909e

  Copy full SHA for a82909e

• Reduce batch

  ikergarcia1996

  committed
  65c53c3

  Copy full SHA for 65c53c3

• Batch size 1

  ikergarcia1996

  committed
  132d17c

  Copy full SHA for 132d17c

• Eval

  ikergarcia1996

  committed
  ff1c372

  Copy full SHA for ff1c372

• Verbosity

  ikergarcia1996

  committed
  6bcd28f

  Copy full SHA for 6bcd28f

• Verbosity

  ikergarcia1996

  committed
  3fde6fb

  Copy full SHA for 3fde6fb

• Verbosity

  ikergarcia1996

  committed
  b6fca26

  Copy full SHA for b6fca26

• Deepspeed launcher

  ikergarcia1996

  committed
  6a93b67

  Copy full SHA for 6a93b67

• 4 bit quantization

  ikergarcia1996

  committed
  9643469

  Copy full SHA for 9643469

• Set low_cpu_mem_usage

  ikergarcia1996

  committed
  b8eef08

  Copy full SHA for b8eef08

• Auto device_map

  ikergarcia1996

  committed
  14adf1c

  Copy full SHA for 14adf1c

• Disable merge and upload

  ikergarcia1996

  committed
  441a8c2

  Copy full SHA for 441a8c2

• Auto device map

  ikergarcia1996

  committed
  52beb8b

  Copy full SHA for 52beb8b

• Run without torchrun

  ikergarcia1996

  committed
  3e5712f

  Copy full SHA for 3e5712f

• Auto devicemap

  ikergarcia1996

  committed
  239d9a2

  Copy full SHA for 239d9a2

• Attempt deepspeed

  ikergarcia1996

  committed
  4b58264

  Copy full SHA for 4b58264

• Auto_devicemap=True

  ikergarcia1996

  committed
  7f31a63

  Copy full SHA for 7f31a63

• Quantization

  ikergarcia1996

  committed
  34d09ee

  Copy full SHA for 34d09ee

• Set 1 batch size

  ikergarcia1996

  committed
  3417370

  Copy full SHA for 3417370

Commits on Dec 13, 2023

• Yi cosine scheduler

  ikergarcia1996

  committed
  781e5a1

  Copy full SHA for 781e5a1

• No quantization

  ikergarcia1996

  committed
  52e7cc6

  Copy full SHA for 52e7cc6

Commits on Dec 15, 2023

• Eval YI

  ikergarcia1996

  committed
  9ef641a

  Copy full SHA for 9ef641a

• Attempt to fix deepspeed inference with LoRA

  ikergarcia1996

  committed
  238ae32

  Copy full SHA for 238ae32

• Launch with accelerate

  ikergarcia1996

  committed
  06d084d

  Copy full SHA for 06d084d

• Run without deepspeed

  ikergarcia1996

  committed
  e74d3e7

  Copy full SHA for e74d3e7

Commits on Apr 24, 2024

• Update load model. Add MOE compatibility. Fix deprecated tests.

  ikergarcia1996

  committed
  fb6ce83

  Copy full SHA for fb6ce83

Commits on May 23, 2024

• Update paraphrase code

  ikergarcia1996

  committed
  594bfe0

  Copy full SHA for 594bfe0

• Fix bug

  ikergarcia1996

  committed
  65a8584

  Copy full SHA for 65a8584

• mgpu

  ikergarcia1996

  committed
  d23e739

  Copy full SHA for d23e739

• Fix

  ikergarcia1996

  committed
  d364556

  Copy full SHA for d364556

• Fix

  ikergarcia1996

  committed
  5026aee

  Copy full SHA for 5026aee

• Fix

  ikergarcia1996

  committed
  a3fd29b

  Copy full SHA for a3fd29b

• Debug

  ikergarcia1996

  committed
  9f38a6f

  Copy full SHA for 9f38a6f

• Disable mgpu

  ikergarcia1996

  committed
  a7e99c2

  Copy full SHA for a7e99c2

• Remove labels if present

  ikergarcia1996

  committed
  dadab39

  Copy full SHA for dadab39

• Fix

  ikergarcia1996

  committed
  200fa48

  Copy full SHA for 200fa48

• Deepspeed and fix output

  ikergarcia1996

  committed
  3768bbd

  Copy full SHA for 3768bbd

• Fix

  ikergarcia1996

  committed
  578d956

  Copy full SHA for 578d956

• Fix

  ikergarcia1996

  committed
  4775307

  Copy full SHA for 4775307

• bf16

  ikergarcia1996

  committed
  a41e4f0

  Copy full SHA for a41e4f0

• Fix decoding

  ikergarcia1996

  committed
  2db45b6

  Copy full SHA for 2db45b6

Commits on May 24, 2024

• Add end to end

  osainz59

  committed
  d3fc9d8

  Copy full SHA for d3fc9d8

• Make style

  osainz59

  committed
  e1b5bdd

  Copy full SHA for e1b5bdd

Commits on May 25, 2024

• add end2end configs

  osainz59

  committed
  63b00e0

  Copy full SHA for 63b00e0

Commits on Jul 24, 2024

• gitignore update

  Neilus03

  committed
  8d1dde1

  Copy full SHA for 8d1dde1

• CHANGES IN DATA GENERATION, NEW LLAMA3 MODEL ADDITION AND GITIGNORE

  Neilus03

  committed
  cb243aa

  Copy full SHA for cb243aa

Commits on Aug 8, 2024

• added updates on run.py and configs for llama3.1

  Neilus03

  committed
  181aff7

  Copy full SHA for 181aff7

• updated run name

  Neilus03

  committed
  ef01604

  Copy full SHA for ef01604

Commits on Aug 26, 2024

• changes in the bash and gitignore files

  Neilus03

  committed
  4983110

  Copy full SHA for 4983110

Commits on Aug 28, 2024

• changes for pretraining on new dataset

  Neilus03

  committed
  2dbadb1

  Copy full SHA for 2dbadb1

Commits on Aug 29, 2024

• Finished sh

  Neilus03

  committed
  b4f5758

  Copy full SHA for b4f5758

• chore: Update GPU resource allocation for pretraining script

  Neilus03

  committed
  30e39bc

  Copy full SHA for 30e39bc

Commits on Aug 30, 2024

• changes in the deepspeed config 2 -> 3 and added task_id mapping

  Neilus03

  committed
  cbadd48

  Copy full SHA for cbadd48

Commits on Sep 2, 2024

• chore: Update GPU resource allocation for pretraining script

  Neilus03

  committed
  d718178

  Copy full SHA for d718178

Commits on Sep 3, 2024

• chore: Update GPU resource allocation and batch sizes for pretraining script

  Neilus03

  committed
  72060ab

  Copy full SHA for 72060ab

Commits on Sep 5, 2024

• Update GPU resource allocation and batch sizes for pretraining script

  Neilus03

  committed
  b901623

  Copy full SHA for b901623

Commits on Sep 6, 2024

• yaml and sh files edited for finetuning od pretrained model

  Neilus03

  committed
  4dd414a

  Copy full SHA for 4dd414a

Commits on Sep 8, 2024

• Update GPU resource allocation and batch sizes for pretraining script

  Neilus03

  committed
  3b095ce

  Copy full SHA for 3b095ce

• update

  Neilus03

  committed
  06de601

  Copy full SHA for 06de601

Commits on Sep 10, 2024

• Update GPU resource allocation and batch sizes for pretraining script

  Neilus03

  committed
  cc57a13

  Copy full SHA for cc57a13

• Update GPU resource allocation and batch sizes for pretraining script

  Neilus03

  committed
  7bca997

  Copy full SHA for 7bca997

Commits on Sep 13, 2024

• chore: frequent logging, model to device, less gpus

  Neilus03

  committed
  47979d3

  Copy full SHA for 47979d3

Commits on Sep 26, 2024

• Refactor typing imports and annotations

  Neilus03

  committed
  7eeaa1e

  Copy full SHA for 7eeaa1e

• CREATED NEW PRETRAINING STRATEGIES

  Neilus03

  committed
  17b86ee

  Copy full SHA for 17b86ee

• DELETE TREE

  Neilus03

  committed
  4abc148

  Copy full SHA for 4abc148

Commits on Sep 29, 2024

• Update pretraining script for masking and curriculum strategies

  Neilus03

  committed
  70ac762

  Copy full SHA for 70ac762

Commits on Sep 30, 2024

• compute results to finetuning_normal_pretrained_vs_finetuning_masked_pretrained.csv

  Neilus03

  committed
  7bbacaf

  Copy full SHA for 7bbacaf

• Add bash script for finetuning with masking and negatives pretrained models

  Neilus03

  committed
  e7cbd27

  Copy full SHA for e7cbd27

• Update pretraining and finetuning config yamls for masking and negatives pretrained models

  Neilus03

  committed
  be209cc

  Copy full SHA for be209cc

• Update config.py for entity type masking and negatives probabilities

  Neilus03

  committed
  bee0286

  Copy full SHA for bee0286

• Apply entity type masking with probability p to JSON entry, todo: negatives

  Neilus03

  committed
  5228b94

  Copy full SHA for 5228b94

• Apply entity type masking and negatives transformations to Collie dataset by creating a new Dataset CollieDatasetWithTransformations

  Neilus03

  committed
  c540296

  Copy full SHA for c540296

• Apply entity type masking and negatives transformations while training

  Neilus03

  committed
  206f269

  Copy full SHA for 206f269

• deleting old unused files

  Neilus03

  committed
  162270d

  Copy full SHA for 162270d

Commits on Oct 3, 2024

• NEGATIVES TO BE MODIFIED YET, NOT WORKING PROPERLY YET

  Neilus03

  committed
  2c6f53b

  Copy full SHA for 2c6f53b

Commits on Oct 11, 2024

• PRETRAINING & FINETUNING BASH SCRIPTS

  Neilus03

  committed
  b714a0f

  Copy full SHA for b714a0f

• PRETRAINING AND FINETUNING CONFIG YAML FILES

  Neilus03

  committed
  bafc96d

  Copy full SHA for bafc96d

• APPLY_NEGATIVES FUNCTION FINISHED AND WORKING

  Neilus03

  committed
  20719e4

  Copy full SHA for 20719e4

Commits on Oct 12, 2024

• negatives loading on dataset.py, old without negatives in dataset_old

  Neilus03

  committed
  d87dbcc

  Copy full SHA for d87dbcc

Commits on Oct 14, 2024

• sentences split correctes and unlabelled text split corrected

  Neilus03

  committed
  b70e38c

  Copy full SHA for b70e38c

• data transformations for negatives fixed, created a function to turn results into a table

  Neilus03

  committed
  3679a41

  Copy full SHA for 3679a41

Commits on Oct 17, 2024

• Fix data transformations for negatives and add table conversion function

  Neilus03

  committed
  1e57c54

  Copy full SHA for 1e57c54

• Fix GPU allocation and memory usage in bash scripts for negatives pretraining

  Neilus03

  committed
  c889a59

  Copy full SHA for c889a59

• created yaml configs for negatives pretraining and finetuning

  Neilus03

  committed
  7d5c69b

  Copy full SHA for 7d5c69b

Commits on Oct 21, 2024

• Fix GPU allocation and memory usage in bash scripts for negatives and curriculum pretraining

  Neilus03

  committed
  4df4db4

  Copy full SHA for 4df4db4

• Refactor dataset class selection and add num_workers argument

  Show description for 07c11ec
  Neilus03

  committed
  07c11ec

  Copy full SHA for 07c11ec

• Refactor dataset class selection to accept the augmentations as a list for curriculum learning possibility

  Neilus03

  committed
  75786d1

  Copy full SHA for 75786d1

• curriculum finetuning configs

  Neilus03

  committed
  13ddf4b

  Copy full SHA for 13ddf4b

• curriculum pretraining configs

  Neilus03

  committed
  a569fc7

  Copy full SHA for a569fc7

Commits on Nov 11, 2024

• 30k pretraining and finetuning yamls and slurms

  Neilus03

  committed
  4da1650

  Copy full SHA for 4da1650

• avoid overfitting in pretraining by doing a single epoch

  Neilus03

  committed
  76b999e

  Copy full SHA for 76b999e

Commits on Nov 12, 2024

• less gpus

  Neilus03

  committed
  555a6e7

  Copy full SHA for 555a6e7

Commits on Nov 13, 2024

• 50k datapoint pretrainging and finetuning setups

  Neilus03

  committed
  7458b45

  Copy full SHA for 7458b45

Commits on Nov 16, 2024

• increased gpus on weekend

  Neilus03

  committed
  89df5c9

  Copy full SHA for 89df5c9

• update finetuning configurations to include pretrained lora weights and add new bash scripts for pretraining and finetuning

  Neilus03

  committed
  d0a3c43

  Copy full SHA for d0a3c43

• update pretraining and finetuning configurations to reflect 30k curriculum and add new bash scripts for 50k training

  Neilus03

  committed
  adebf1f

  Copy full SHA for adebf1f

Commits on Nov 17, 2024

• add finetuning configuration for 50k pretrained model with updated training and evaluation settings

  Neilus03

  committed
  6fcf424

  Copy full SHA for 6fcf424

• add new bash scripts for 50k pretraining and finetuning configurations with updated model settings

  Neilus03

  committed
  760fa50

  Copy full SHA for 760fa50

Commits on Nov 18, 2024

• less gpus monday

  Neilus03

  committed
  c112730

  Copy full SHA for c112730

Commits on Nov 25, 2024

• add evaluation scripts and configuration files for Llama3 model pretraining and evaluation on gollie test sets

  Neilus03

  committed
  04881ae

  Copy full SHA for 04881ae

• remove unused pretraining evaluation configuration for GoLLIE-8B Llama3 model

  Neilus03

  committed
  d6c00d7

  Copy full SHA for d6c00d7

• add evaluation script and configuration for 30k pretraining of Llama3 model

  Neilus03

  committed
  874559b

  Copy full SHA for 874559b

Commits on Nov 27, 2024

• update pretraining script to use 2 GPUs and add new evaluation script and configuration for 50k pretraining of Llama3 model

  Neilus03

  committed
  674938a

  Copy full SHA for 674938a

Commits on Dec 2, 2024

• reduce GPU allocation from 4 to 2 for 30k and 50k finetuning scripts

  Neilus03

  committed
  b51f16e

  Copy full SHA for b51f16e

Commits on Dec 4, 2024

• add finetuning and pretraining scripts for GoLLIE-8B and Llama3 models; adjust GPU allocation for 40k evaluation

  Neilus03

  committed
  502ac38

  Copy full SHA for 502ac38

Commits on Dec 6, 2024

• add function to save merged LoRA model and tokenizer; implement logic for merging before inference

  Neilus03

  committed
  6b16a92

  Copy full SHA for 6b16a92

• add configuration files for GoLLIE-8B and Llama3 models; include finetuning and pretraining settings for 40k evaluation

  Neilus03

  committed
  238aaec

  Copy full SHA for 238aaec

Commits on Jan 20, 2025

• update finetuning configuration for GoLLIE-8B and Llama3 models; ensure proper torch_dtype formatting

  Neilus03

  committed
  4bb2a0a

  Copy full SHA for 4bb2a0a

Commits on Jan 22, 2025

• remove unnecessary comments and import functions from data_transformations in dataset.py

  Neilus03

  committed
  10bb8ba

  Copy full SHA for 10bb8ba