Code

[muP] Rework#1087

Code

Open

lintangsutawika wants to merge 109 commits intomainEleutherAI/gpt-neox:mainfrom rework-mupEleutherAI/gpt-neox:rework-mupCopy head branch name to clipboard

+683-627

Commits

Commits on Dec 1, 2023

• changed ordering for setting up norm_factor

  lintangsutawika

  committed
  0d921f7

  Copy full SHA for 0d921f7

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  abee54d

  Copy full SHA for abee54d

• updated muP args to the minimum required

  lintangsutawika

  committed
  a08c3ef

  Copy full SHA for a08c3ef

• calculate m_width

  lintangsutawika

  committed
  c35e830

  Copy full SHA for c35e830

• Merge branch 'main' of https://github.com/EleutherAI/gpt-neox into rework-mup

  lintangsutawika

  committed
  2807e52

  Copy full SHA for 2807e52

• Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox into rework-mup

  lintangsutawika

  committed
  2d127df

  Copy full SHA for 2d127df

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  81fdc4d

  Copy full SHA for 81fdc4d

• changed ordering for setting up norm_factor

  lintangsutawika

  committed
  7d6b246

  Copy full SHA for 7d6b246

• updated muP args to the minimum required

  lintangsutawika

  committed
  a0d1929

  Copy full SHA for a0d1929

• calculate m_width

  lintangsutawika

  committed
  d63b3b8

  Copy full SHA for d63b3b8

• Update NeoXArgs docs automatically

  

  github-actions

  authored and

  lintangsutawika

  committed
  9be82fe

  Copy full SHA for 9be82fe

• removed redundant line

  lintangsutawika

  committed
  66214d9

  Copy full SHA for 66214d9

• removed redundant lines

  lintangsutawika

  committed
  17b7183

  Copy full SHA for 17b7183

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  a6bad07

  Copy full SHA for a6bad07

• removed redundant lines

  lintangsutawika

  committed
  63984bd

  Copy full SHA for 63984bd

• Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox into rework-mup

  lintangsutawika

  committed
  02687a8

  Copy full SHA for 02687a8

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  11114e2

  Copy full SHA for 11114e2

• modify init with mup

  lintangsutawika

  committed
  05c4de3

  Copy full SHA for 05c4de3

• divide logits by the m_width

  lintangsutawika

  committed
  71a91e4

  Copy full SHA for 71a91e4

• moved position of mup parameters being processed

  lintangsutawika

  committed
  99c8ce0

  Copy full SHA for 99c8ce0

• add note

  lintangsutawika

  committed
  b253ab6

  Copy full SHA for b253ab6

Commits on Dec 6, 2023

• made param groups to hold flag for mup scaling

  lintangsutawika

  committed
  1919499

  Copy full SHA for 1919499

• lr scale

  lintangsutawika

  committed
  17678e0

  Copy full SHA for 17678e0

• update config

  lintangsutawika

  committed
  2bd5ae6

  Copy full SHA for 2bd5ae6

• adjust process of mup variables

  lintangsutawika

  committed
  6642291

  Copy full SHA for 6642291

Commits on Dec 18, 2023

• remove calling save_base_shapes

  lintangsutawika

  committed
  8be6c66

  Copy full SHA for 8be6c66

• lr adjustments is done in train_step to address lr being reset due to lr_scheduling

  lintangsutawika

  committed
  c9fb18b

  Copy full SHA for c9fb18b

• lr scaling for mup is moved here instead

  lintangsutawika

  committed
  795371c

  Copy full SHA for 795371c

Commits on Jan 3, 2024

• removed mup usage for coord check

  lintangsutawika

  committed
  087beee

  Copy full SHA for 087beee

• merged with main

  lintangsutawika

  committed
  16d04b1

  Copy full SHA for 16d04b1

Commits on Jan 24, 2024

• latest update on coord check implementation

  lintangsutawika

  committed
  e7b7bf6

  Copy full SHA for e7b7bf6

Commits on Feb 2, 2024

• fix merge conflict

  lintangsutawika

  committed
  8dea9ce

  Copy full SHA for 8dea9ce

• changed mup_m_width to mup_width_multiplier

  lintangsutawika

  committed
  3664eba

  Copy full SHA for 3664eba

• fixed notations

  lintangsutawika

  committed
  6a46247

  Copy full SHA for 6a46247

• correct scale

  lintangsutawika

  committed
  7439f9a

  Copy full SHA for 7439f9a

• m_emb * embed(X)

  lintangsutawika

  committed
  5b2d31c

  Copy full SHA for 5b2d31c

• removed mup rescale in the layers

  lintangsutawika

  committed
  98caa82

  Copy full SHA for 98caa82

• removed mup rescale in the layers

  lintangsutawika

  committed
  5c99637

  Copy full SHA for 5c99637

• adjust mup_m_emb to mup_embedding_multiplier

  lintangsutawika

  committed
  a636f06

  Copy full SHA for a636f06

Commits on Feb 20, 2024

• add multiplier mup_output_multiplier

  lintangsutawika

  committed
  39190c5

  Copy full SHA for 39190c5

• reorder model loading

  lintangsutawika

  committed
  2489cc0

  Copy full SHA for 2489cc0

• removed comments

  lintangsutawika

  committed
  23b8776

  Copy full SHA for 23b8776

• removed comments

  lintangsutawika

  committed
  10e935e

  Copy full SHA for 10e935e

• implement full process

  lintangsutawika

  committed
  a0aca99

  Copy full SHA for a0aca99

Commits on Feb 21, 2024

• set neox_args.iteration to 0 for coord_check mode

  lintangsutawika

  committed
  9472b35

  Copy full SHA for 9472b35

• move mup_width_multiplier init

  lintangsutawika

  committed
  5c5f2df

  Copy full SHA for 5c5f2df

• mup_coord_check returns 2 df

  lintangsutawika

  committed
  7eca3e7

  Copy full SHA for 7eca3e7

• can run

  lintangsutawika

  committed
  c9a3a65

  Copy full SHA for c9a3a65

Commits on Feb 22, 2024

• remove commehts

  lintangsutawika

  committed
  a7877d4

  Copy full SHA for a7877d4

• add hooks

  lintangsutawika

  committed
  bd9d399

  Copy full SHA for bd9d399

• remove comments

  lintangsutawika

  committed
  fe180d3

  Copy full SHA for fe180d3

• uncomment activation data

  lintangsutawika

  committed
  b240c19

  Copy full SHA for b240c19

• plot coords

  lintangsutawika

  committed
  93b4241

  Copy full SHA for 93b4241

• removed variables, add way to plot only from rank 0

  lintangsutawika

  committed
  d4899fc

  Copy full SHA for d4899fc

• changed key name in dict

  lintangsutawika

  committed
  f589e29

  Copy full SHA for f589e29

• remove print

  lintangsutawika

  committed
  8261e0d

  Copy full SHA for 8261e0d

• fix how width_multiplier is applied

  lintangsutawika

  committed
  25aa786

  Copy full SHA for 25aa786

• updated plot config

  lintangsutawika

  committed
  4d246a1

  Copy full SHA for 4d246a1

Commits on Feb 26, 2024

• update files

  lintangsutawika

  committed
  84c5380

  Copy full SHA for 84c5380

• Merge branch 'main' into rework-mup

  lintangsutawika

  authored
  b2f1101

  Copy full SHA for b2f1101

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  42d4cde

  Copy full SHA for 42d4cde

Commits on Feb 27, 2024

• init function, add input embedding different initialization

  lintangsutawika

  committed
  4c477d5

  Copy full SHA for 4c477d5

• Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox into rework-mup

  lintangsutawika

  committed
  64dc4c5

  Copy full SHA for 64dc4c5

• changeoutput layer to normal

  lintangsutawika

  committed
  65c103e

  Copy full SHA for 65c103e

• change from mean to std

  lintangsutawika

  committed
  08b5d40

  Copy full SHA for 08b5d40

• double attention head for every hidden size doubled

  lintangsutawika

  committed
  2ca94a8

  Copy full SHA for 2ca94a8

• Merge branch 'main' into rework-mup

  lintangsutawika

  authored
  7483246

  Copy full SHA for 7483246

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  497485c

  Copy full SHA for 497485c

• added args

  lintangsutawika

  committed
  34fb7ca

  Copy full SHA for 34fb7ca

• simplify coordcheck

  lintangsutawika

  committed
  2d53f1f

  Copy full SHA for 2d53f1f

• seperate sp and mup configs

  lintangsutawika

  committed
  7897610

  Copy full SHA for 7897610

• perform coordcheck for sp and mup seperately

  lintangsutawika

  committed
  4f39209

  Copy full SHA for 4f39209

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  5f84a3f

  Copy full SHA for 5f84a3f

Commits on Feb 28, 2024

• update

  lintangsutawika

  committed
  479b854

  Copy full SHA for 479b854

• update how params are sorted

  lintangsutawika

  committed
  21a7e32

  Copy full SHA for 21a7e32

• remove unused comments

  lintangsutawika

  committed
  bb2e0c9

  Copy full SHA for bb2e0c9

Commits on Feb 29, 2024

• adjust

  lintangsutawika

  committed
  bf1ce06

  Copy full SHA for bf1ce06

• simplify

  lintangsutawika

  committed
  50a3dba

  Copy full SHA for 50a3dba

• fix mup embedding multiplier

  lintangsutawika

  committed
  c4c1660

  Copy full SHA for c4c1660

• embeddingpipe fix init

  lintangsutawika

  committed
  1c35911

  Copy full SHA for 1c35911

• changed how manual seed is loaded

  lintangsutawika

  committed
  84be4d4

  Copy full SHA for 84be4d4

• removed musgd and other changces

  lintangsutawika

  committed
  fbb4daf

  Copy full SHA for fbb4daf

• update config

  lintangsutawika

  committed
  fa142ff

  Copy full SHA for fa142ff

• fixed how params are sorted

  lintangsutawika

  committed
  ad2336f

  Copy full SHA for ad2336f

• update how seed is computed

  lintangsutawika

  committed
  fe73bc3

  Copy full SHA for fe73bc3

• update to follow pre-commit format

  lintangsutawika

  committed
  a3bd44c

  Copy full SHA for a3bd44c

• update from main

  lintangsutawika

  committed
  56b6c9b

  Copy full SHA for 56b6c9b

• update

  lintangsutawika

  committed
  2365fd5

  Copy full SHA for 2365fd5

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  e8639a0

  Copy full SHA for e8639a0

Commits on Mar 5, 2024

• fix lr weighting

  lintangsutawika

  committed
  47e1438

  Copy full SHA for 47e1438

• hard set to 1.0 if neox_args.use_mup is false

  lintangsutawika

  committed
  a064f9b

  Copy full SHA for a064f9b

Commits on Apr 21, 2024

• Merge branch 'main' into rework-mup

  Quentin-Anthony

  authored
  b0da27a

  Copy full SHA for b0da27a

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  6fe55f4

  Copy full SHA for 6fe55f4

Commits on May 2, 2024

• add new parameters

  lintangsutawika

  committed
  8bf8bcd

  Copy full SHA for 8bf8bcd

• add parameter checks

  lintangsutawika

  committed
  7f0b033

  Copy full SHA for 7f0b033

• updates to argument processing for mup

  lintangsutawika

  committed
  f802869

  Copy full SHA for f802869

• add data save and descriptions being printed

  lintangsutawika

  committed
  cc71104

  Copy full SHA for cc71104

• update mup

  lintangsutawika

  committed
  c8feb39

  Copy full SHA for c8feb39

• update seed

  lintangsutawika

  committed
  b6b3a02

  Copy full SHA for b6b3a02

• remove print text

  lintangsutawika

  committed
  847e892

  Copy full SHA for 847e892

• fixed kv

  lintangsutawika

  committed
  1b0027c

  Copy full SHA for 1b0027c

• update

  lintangsutawika

  committed
  055596f

  Copy full SHA for 055596f

• update dewcriptions being printed

  lintangsutawika

  committed
  fabb45b

  Copy full SHA for fabb45b

• removed unused lines

  lintangsutawika

  committed
  5ccf693

  Copy full SHA for 5ccf693

• Merge branch 'rework-mup' of https://github.com/EleutherAI/gpt-neox into rework-mup

  lintangsutawika

  committed
  9dd583b

  Copy full SHA for 9dd583b

• Merge branch 'main' into rework-mup

  lintangsutawika

  authored
  6a8ad71

  Copy full SHA for 6a8ad71

• Update NeoXArgs docs automatically

  

  github-actions

  committed
  485cad4

  Copy full SHA for 485cad4

Commits on Sep 23, 2024

• Merge branch 'main' into rework-mup

  Quentin-Anthony

  authored
  c291906

  Copy full SHA for c291906

Commits on Oct 8, 2024

• Merge branch 'main' into rework-mup

  Quentin-Anthony

  authored
  1ac9add

  Copy full SHA for 1ac9add