Code

Ablation#2

Code

Open

Jeronymous wants to merge 41 commits intomasterOpenLLM-France/Lucie-Training:masterfrom ablationOpenLLM-France/Lucie-Training:ablationCopy head branch name to clipboard

+2,232-4

Commits

Commits on Aug 21, 2024

• feat: add perplexity scripts

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  e20cc7d

  Copy full SHA for e20cc7d

• ablation: update data collection script to adapt to the new domains

  Agustin-Picard

  committed
  10a39d2

  Copy full SHA for 10a39d2

• ablation: introduce the domain proportions in yaml format

  Agustin-Picard

  committed
  bd8586a

  Copy full SHA for bd8586a

• ablation: update script to call the right python script for the domains' weights

  Agustin-Picard

  committed
  2da1c99

  Copy full SHA for 2da1c99

• ablation: modify eval function to return results dict instead of file

  Agustin-Picard

  committed
  2bd1627

  Copy full SHA for 2bd1627

• ablation: turn off DP as it was causing trouble during GPU sync

  Agustin-Picard

  committed
  cba8dc9

  Copy full SHA for cba8dc9

• - Add json files with raw statistics.

  Show description for 226e131
  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  226e131

  Copy full SHA for 226e131

• Add category columns to dataset

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  a693922

  Copy full SHA for a693922

• Fix Pile categories

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  a8d6a75

  Copy full SHA for a8d6a75

• Update stats (fix CroissantAligned/detailed, update Gallica and Gutenberg)

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  a2eee0c

  Copy full SHA for a2eee0c

• Discard MathPile validation set for training

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  2526f63

  Copy full SHA for 2526f63

• add counts for categories, ocr, lang

  

  jhunter19

  authored and

  Agustin-Picard

  committed
  f7233a9

  Copy full SHA for f7233a9

• Update token statistics, and statistics about YouTube

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  5d85e05

  Copy full SHA for 5d85e05

• Breakdown Pile and OtherFr

  

  Jeronymous

  authored and

  Agustin-Picard

  committed
  899745a

  Copy full SHA for 899745a

• Update categories

  

  jhunter19

  authored and

  Agustin-Picard

  committed
  da8f62a

  Copy full SHA for da8f62a

• feat: update for domain and language

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  7f878af

  Copy full SHA for 7f878af

• fix: minor mistakes with paths and unused variables

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  9255142

  Copy full SHA for 9255142

• feat: change the domain proportion yaml file to integrate the language

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  70e50f7

  Copy full SHA for 70e50f7

• ablation: Fix DP in PPL evaluation

  Agustin-Picard

  committed
  b7c3031

  Copy full SHA for b7c3031

• feat: agus add distributing arguments fr correct computation

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  a3be345

  Copy full SHA for a3be345

• fix: the domain proportion for programming language was not normalized

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  1e0ee64

  Copy full SHA for 1e0ee64

• feat: add a slurm file for launching ablation for the 80M model

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  e015a85

  Copy full SHA for e015a85

• fixup: forget a reflink to home

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  06b4483

  Copy full SHA for 06b4483

• fix: a broken link

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  00d2c49

  Copy full SHA for 00d2c49

• fix: remove a print which break the script with CulturaX

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  646247f

  Copy full SHA for 646247f

• feat: automatically set the output path to the user work directory

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  e74fe67

  Copy full SHA for e74fe67

• feat: add a slurm training job for 80 M parameters model

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  900f230

  Copy full SHA for 900f230

• fix: wrong print rank

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  c7c1f3e

  Copy full SHA for c7c1f3e

• feat: improve ablation study experiments management

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  7a779d3

  Copy full SHA for 7a779d3

• chore: ease the launch of multiple experiments

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  49fde71

  Copy full SHA for 49fde71

• fix: typo mistake

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  a0c50e8

  Copy full SHA for a0c50e8

• fix: update the splits so there is no empty datasets

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  4bc9cdf

  Copy full SHA for 4bc9cdf

• fix: write numpy array in the csv instead of torch tensor

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  9a4fade

  Copy full SHA for 9a4fade

• feat: avoid the drop last batch as some test samples are not large enough for that

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  2ecd99e

  Copy full SHA for 2ecd99e

• feat: change the data loader

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  ec35b26

  Copy full SHA for ec35b26

• fix: wrong wild card to move the outputs files

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  1438cd8

  Copy full SHA for 1438cd8

• feat: add the tokenizers parallelism variable

  

  lucashervier

  authored and

  Agustin-Picard

  committed
  4c9bba5

  Copy full SHA for 4c9bba5

• ablation: update bash scripts to point to correct folders (for me)

  Agustin-Picard

  committed
  03af559

  Copy full SHA for 03af559

• ablation: update data config file to contain a weight for web datasets

  Agustin-Picard

  committed
  8058464

  Copy full SHA for 8058464

• ablation: initialize a script to analyze results from the experiments (unfinished)

  Agustin-Picard

  committed
  f904967

  Copy full SHA for f904967

• chore: clean-up a conflict from the rebase

  Agustin-Picard

  committed
  8b5cec0

  Copy full SHA for 8b5cec0