Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dev neil branch creation #23

Open
wants to merge 181 commits into
base: main
Choose a base branch
from
Open

Dev neil branch creation #23

wants to merge 181 commits into from

Conversation

Neilus03
Copy link

No description provided.

ikergarcia1996 and others added 30 commits October 20, 2023 13:15
@ikergarcia1996
Use HF flash attention and test new hparams
8051dc3
@ikergarcia1996
Merge pull request #1 from hitz-zentroa/main
dc8d8c1 
Rebase to main
@ikergarcia1996
Change name
6a857c3
@ikergarcia1996
Merge remote-tracking branch 'origin/dev' into dev
4ce6f63
@ikergarcia1996
Set correct environment
1590daa
@ikergarcia1996
Change hparams
cf99869
@ikergarcia1996
2 GPUs
5a10fbe
@ikergarcia1996
change project
38d51c1
@ikergarcia1996
Fix bug
d2df066
@ikergarcia1996
128
2a462ee
@ikergarcia1996
Fix name
42e95a5
@ikergarcia1996
fix memory fragmentation
c9f08f4
@ikergarcia1996
Reduce batch size
ac41590
@ikergarcia1996
Use 4 GPUs
d736865
@ikergarcia1996
Increase num threads
9ab89a8
@ikergarcia1996
Evaluate all checkpoints
4574408
@ikergarcia1996
Attempt to fix dtype bug
3495f41
@ikergarcia1996
Fix eval scripts
4dabec4
@ikergarcia1996
Fix names
f579417
@ikergarcia1996
Marge model before inference
9c813b2
@ikergarcia1996
Merge models before inference
a309f25
@ikergarcia1996
Fix bug
8ffa1c5
@ikergarcia1996
bf16 eval
d809598
@ikergarcia1996
Disable flash attention
7d58fa7
@ikergarcia1996
Prepare run 128
4c3e290
@ikergarcia1996
Set correct output path
d6d4b52
@ikergarcia1996
Eval 256
23d4fd0
@ikergarcia1996
Fix path
eba9e1f
@ikergarcia1996
Use flash attention
ec4536a
@ikergarcia1996
Eval 128
8cd3732
Neilus03 added 30 commits October 14, 2024 16:32
@Neilus03
sentences split correctes and unlabelled text split corrected
b70e38c
@Neilus03
data transformations for negatives fixed, created a function to turn …
3679a41 
…results into a table
@Neilus03
Fix data transformations for negatives and add table conversion function
1e57c54
@Neilus03
Fix GPU allocation and memory usage in bash scripts for negatives pre…
c889a59 
…training
@Neilus03
created yaml configs for negatives pretraining and finetuning
7d5c69b
@Neilus03
Fix GPU allocation and memory usage in bash scripts for negatives and…
4df4db4 
… curriculum pretraining
@Neilus03
Refactor dataset class selection and add num_workers argument
07c11ec 
This commit refactors the dataset class selection in the `train_collie` function. If either `data_args.entity_type_masking_prob` or `data_args.negatives_prob` is set, the `CollieDatasetWithTransformations` class is used. Otherwise, the base `CollieDataset` class is used. Additionally, the `num_workers` argument is added to limit the number of workers to the minimum of the available CPU count and 16.
@Neilus03
Refactor dataset class selection to accept the augmentations as a lis…
75786d1 
…t for curriculum learning possibility
@Neilus03
curriculum finetuning configs
13ddf4b
@Neilus03
curriculum pretraining configs
a569fc7
@Neilus03
30k pretraining and finetuning yamls and slurms
4da1650
@Neilus03
avoid overfitting in pretraining by doing a single epoch
76b999e
@Neilus03
less gpus
555a6e7
@Neilus03
50k datapoint pretrainging and finetuning setups
7458b45
@Neilus03
increased gpus on weekend
89df5c9
@Neilus03
update finetuning configurations to include pretrained lora weights a…
d0a3c43 
…nd add new bash scripts for pretraining and finetuning
@Neilus03
update pretraining and finetuning configurations to reflect 30k curri…
adebf1f 
…culum and add new bash scripts for 50k training
@Neilus03
add finetuning configuration for 50k pretrained model with updated tr…
6fcf424 
…aining and evaluation settings
@Neilus03
add new bash scripts for 50k pretraining and finetuning configuration…
760fa50 
…s with updated model settings
@Neilus03
less gpus monday
c112730
@Neilus03
add evaluation scripts and configuration files for Llama3 model pretr…
04881ae 
…aining and evaluation on gollie test sets
@Neilus03
remove unused pretraining evaluation configuration for GoLLIE-8B Llam…
d6c00d7 
…a3 model
@Neilus03
add evaluation script and configuration for 30k pretraining of Llama3…
874559b 
… model
@Neilus03
update pretraining script to use 2 GPUs and add new evaluation script…
674938a 
… and configuration for 50k pretraining of Llama3 model
@Neilus03
reduce GPU allocation from 4 to 2 for 30k and 50k finetuning scripts
b51f16e
@Neilus03
add finetuning and pretraining scripts for GoLLIE-8B and Llama3 model…
502ac38 
…s; adjust GPU allocation for 40k evaluation
@Neilus03
add function to save merged LoRA model and tokenizer; implement logic…
6b16a92 
… for merging before inference
@Neilus03
add configuration files for GoLLIE-8B and Llama3 models; include fine…
238aaec 
…tuning and pretraining settings for 40k evaluation
@Neilus03
update finetuning configuration for GoLLIE-8B and Llama3 models; ensu…
4bb2a0a 
…re proper torch_dtype formatting
@Neilus03
remove unnecessary comments and import functions from data_transforma…
10bb8ba 
…tions in dataset.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Neilus03 @ikergarcia1996 @osainz59