Code

Idefics3-style VLM training support#24

Code

Open

kisate wants to merge 58 commits intoswiss-ai:mainswiss-ai/nanotron:mainfrom kisate:mainkisate/nanotron:mainCopy head branch name to clipboard

+5,639-119

Commits

Commits on Oct 23, 2024

• Idefics2 implementation [untested]

  kisate

  committed
  96b8d92

  Copy full SHA for 96b8d92

• small fixes

  kisate

  committed
  f57196c

  Copy full SHA for f57196c

Commits on Oct 27, 2024

• Siglip weights convertation runs

  kisate

  committed
  682c74b

  Copy full SHA for 682c74b

Commits on Oct 30, 2024

• Shift to idefics3 (initial)

  kisate

  committed
  1e8fd10

  Copy full SHA for 1e8fd10

• rename idefics2 to idefics3

  kisate

  committed
  ad77aa3

  Copy full SHA for ad77aa3

• hf vs nanotron idefics3

  kisate

  committed
  b41f0f3

  Copy full SHA for b41f0f3

• Constructed idefics3 runs

  kisate

  committed
  f6bcfe3

  Copy full SHA for f6bcfe3

• Convert from hf produces working weights

  kisate

  committed
  12a71c1

  Copy full SHA for 12a71c1

Commits on Nov 11, 2024

• Fixed attention calculation differences

  kisate

  committed
  f3986c8

  Copy full SHA for f3986c8

• Fixed vision transformer outputs

  kisate

  committed
  021189c

  Copy full SHA for 021189c

Commits on Nov 25, 2024

• tmp dataloader

  kisate

  committed
  e23661f

  Copy full SHA for e23661f

• debugging memory usage

  

  kisate

  committed
  83320e6

  Copy full SHA for 83320e6

• fixing dataloader

  kisate

  committed
  c3d2a2b

  Copy full SHA for c3d2a2b

Commits on Dec 3, 2024

• fix vqa collator

  kisate

  committed
  0f73937

  Copy full SHA for 0f73937

• Fix collating

  kisate

  committed
  6e6cb8d

  Copy full SHA for 6e6cb8d

• Training fix

  kisate

  committed
  6c1c20a

  Copy full SHA for 6c1c20a

• Training fix

  kisate

  committed
  8e76020

  Copy full SHA for 8e76020

Commits on Dec 4, 2024

• Metadata load crutch

  kisate

  committed
  11f8561

  Copy full SHA for 11f8561

• Switch idefics3 to all reduce

  kisate

  committed
  0e258fc

  Copy full SHA for 0e258fc

• config for cluster

  kisate

  committed
  7f249c1

  Copy full SHA for 7f249c1

• Merge branch 'dev' of https://github.com/kisate/nanotron into dev

  kisate

  committed
  645df44

  Copy full SHA for 645df44

• Remove prints

  kisate

  committed
  22c9ff1

  Copy full SHA for 22c9ff1

• Update builder script

  kisate

  committed
  2847ef6

  Copy full SHA for 2847ef6

Commits on Dec 5, 2024

• Reduce scattter fixed

  kisate

  committed
  de50df8

  Copy full SHA for de50df8

• Idefics3 reduce scatter support

  kisate

  committed
  eea167c

  Copy full SHA for eea167c

• Standalone ViT

  kisate

  committed
  5ef0c0e

  Copy full SHA for 5ef0c0e

Commits on Dec 7, 2024

• WIP modular iterable dataloader

  kisate

  committed
  d11c260

  Copy full SHA for d11c260

• Iterable dataset testing

  kisate

  committed
  6af18b1

  Copy full SHA for 6af18b1

• Optimizing iterable dataset

  kisate

  committed
  45f79a5

  Copy full SHA for 45f79a5

Commits on Dec 8, 2024

• Move collating to dataloader

  kisate

  committed
  61b235c

  Copy full SHA for 61b235c

• Fix Idefics3 PP

  kisate

  committed
  00b5742

  Copy full SHA for 00b5742

• Idefics3 PP debug

  kisate

  committed
  683cd8a

  Copy full SHA for 683cd8a

• Combine embeddings into a single pipeline block

  kisate

  committed
  126b601

  Copy full SHA for 126b601

Commits on Dec 9, 2024

• Reoder modules in init

  kisate

  committed
  fccd8ef

  Copy full SHA for fccd8ef

• Create convert_nanotron_to_hf.py

  eryaguo

  authored
  994dcb9

  Copy full SHA for 994dcb9

Commits on Dec 11, 2024

• Added idefics3 testing scripts

  

  kisate

  committed
  2b363b5

  Copy full SHA for 2b363b5

Commits on Dec 15, 2024

• preprocessed dataset & loading for ckpt for iterable dataset

  kisate

  committed
  ea46a99

  Copy full SHA for ea46a99

• Merge branch 'dev' of https://github.com/kisate/nanotron into dev

  kisate

  committed
  99b8450

  Copy full SHA for 99b8450

Commits on Dec 18, 2024

• Merge pull request #1 from eryaguo/patch-1

  Show description for 9648f7f
  kisate

  authored
  9648f7f

  Copy full SHA for 9648f7f

• Update to hf conversion script

  kisate

  committed
  25e2a2c

  Copy full SHA for 25e2a2c

• Formatting & comments

  kisate

  committed
  9b9be2e

  Copy full SHA for 9b9be2e

• Cleanup

  kisate

  committed
  8d14715

  Copy full SHA for 8d14715

• Format imports

  kisate

  committed
  961b6c6

  Copy full SHA for 961b6c6

• Extract vqa from dataloader.py

  kisate

  committed
  6fd81c3

  Copy full SHA for 6fd81c3

• Formatting

  kisate

  committed
  0d0c7c3

  Copy full SHA for 0d0c7c3

• Merge branch 'dev' of https://github.com/kisate/nanotron into dev

  kisate

  committed
  2dc9d4d

  Copy full SHA for 2dc9d4d

• Merge branch 'dev' of https://github.com/kisate/nanotron into dev

  kisate

  committed
  d9555e5

  Copy full SHA for d9555e5

• Add image dataset to the main run_train.py

  kisate

  committed
  5968148

  Copy full SHA for 5968148

Commits on Dec 19, 2024

• Restore lost changes

  kisate

  committed
  d6baf07

  Copy full SHA for d6baf07

• Add README to tools & update paths

  kisate

  committed
  c41a099

  Copy full SHA for c41a099

• Merge commit '5968148cb863f44fc40386c1719b55eeea2e51c4' into HEAD

  kisate

  committed
  00bfbe4

  Copy full SHA for 00bfbe4

• Merge pull request #2 from kisate/dev2

  Show description for 5f0046a
  kisate

  authored
  5f0046a

  Copy full SHA for 5f0046a

• Update building script

  kisate

  committed
  dceee54

  Copy full SHA for dceee54

• Verify main scripts

  kisate

  committed
  c4b9f6b

  Copy full SHA for c4b9f6b

• Delete unneded files

  kisate

  committed
  49ec1cd

  Copy full SHA for 49ec1cd

• Readme & cleanup

  kisate

  committed
  db92483

  Copy full SHA for db92483

• Small doc changes

  kisate

  committed
  e83c52e

  Copy full SHA for e83c52e

• Merge pull request #3 from kisate/dev

  Show description for 4ac6fa6
  kisate

  authored
  4ac6fa6

  Copy full SHA for 4ac6fa6