Code

WIP: Adding Logging #33

Code

Closed

mvanniasingheTT wants to merge 78 commits intomaintenstorrent/tt-inference-server:mainfrom mvanniasinghe/loggingtenstorrent/tt-inference-server:mvanniasinghe/loggingCopy head branch name to clipboard

+396-9

Commits

Commits on Oct 22, 2024

• added mock model

  mvanniasingheTT

  committed
  36f5b77

  Copy full SHA for 36f5b77

Commits on Oct 24, 2024

• adding vLLM dockerfile

  tstescoTT

  committed
  ee9be29

  Copy full SHA for ee9be29

• adding label GHCR and sed import edit for vllm server_example_tt.py

  tstescoTT

  committed
  efe648b

  Copy full SHA for efe648b

• adding evals instructions and run script

  tstescoTT

  committed
  0e5f431

  Copy full SHA for 0e5f431

• move vllm llama 3.1 70b implementation to top level model impl dir

  tstescoTT

  committed
  ffdd77a

  Copy full SHA for ffdd77a

• update eval instructions

  tstescoTT

  committed
  5ef0d64

  Copy full SHA for 5ef0d64

• adding llama 3.1 70b benchmarking instructions

  tstescoTT

  committed
  33340dc

  Copy full SHA for 33340dc

• adding dir for locust

  tstescoTT

  committed
  50626e4

  Copy full SHA for 50626e4

• add doc link for vllm setup

  tstescoTT

  committed
  11c604a

  Copy full SHA for 11c604a

• adding pre-commit with ruff linting and formatting

  tstescoTT

  committed
  45f544d

  Copy full SHA for 45f544d

• add pre-commit instructions

  tstescoTT

  committed
  64281b0

  Copy full SHA for 64281b0

• add GHCR repo connection label to Dockerfile

  tstescoTT

  committed
  f9b4fb3

  Copy full SHA for f9b4fb3

• created + add mock model and mock offline inference w/ patches

  mvanniasingheTT

  committed
  3efd4c6

  Copy full SHA for 3efd4c6

• removed unneeded imports from mock model

  mvanniasingheTT

  committed
  f2b8423

  Copy full SHA for f2b8423

• Merge branch 'tstesco/evals-benchmarking-structure' into mvanniasinghe/mock_model

  mvanniasingheTT

  committed
  438e3ae

  Copy full SHA for 438e3ae

• adding vLLM dockerfile

  tstescoTT

  committed
  b687ace

  Copy full SHA for b687ace

• adding label GHCR and sed import edit for vllm server_example_tt.py

  tstescoTT

  committed
  ed34e60

  Copy full SHA for ed34e60

• adding evals instructions and run script

  tstescoTT

  committed
  fd0c87d

  Copy full SHA for fd0c87d

• move vllm llama 3.1 70b implementation to top level model impl dir

  tstescoTT

  committed
  4d0cb6b

  Copy full SHA for 4d0cb6b

• update eval instructions

  tstescoTT

  committed
  a793d35

  Copy full SHA for a793d35

• adding llama 3.1 70b benchmarking instructions

  tstescoTT

  committed
  7ed83b1

  Copy full SHA for 7ed83b1

• adding dir for locust

  tstescoTT

  committed
  0e19de6

  Copy full SHA for 0e19de6

• add doc link for vllm setup

  tstescoTT

  committed
  4f7a328

  Copy full SHA for 4f7a328

• adding pre-commit with ruff linting and formatting

  tstescoTT

  committed
  b40c536

  Copy full SHA for b40c536

• add pre-commit instructions

  tstescoTT

  committed
  488c19c

  Copy full SHA for 488c19c

• add GHCR repo connection label to Dockerfile

  tstescoTT

  committed
  ee888d9

  Copy full SHA for ee888d9

• update python path to include vllm

  mvanniasingheTT

  committed
  5a0a124

  Copy full SHA for 5a0a124

Commits on Oct 25, 2024

• adding requirements-dev.txt

  tstescoTT

  committed
  6c35b17

  Copy full SHA for 6c35b17

• adding EOF newlines

  tstescoTT

  committed
  022cd77

  Copy full SHA for 022cd77

• add ifeval to run_evals.sh, set to stream=False

  tstescoTT

  committed
  f17a922

  Copy full SHA for f17a922

• uncommetned sys path line

  mvanniasingheTT

  committed
  3b48006

  Copy full SHA for 3b48006

• update dockerfile to pip install for specifc commit of vllm

  mvanniasingheTT

  committed
  c1b152c

  Copy full SHA for c1b152c

• update readme mock instructions

  mvanniasingheTT

  committed
  291e872

  Copy full SHA for 291e872

• update readme mock instructions

  mvanniasingheTT

  committed
  ddbb7d4

  Copy full SHA for ddbb7d4

• pin dev dependency versions

  tstescoTT

  committed
  443e930

  Copy full SHA for 443e930

• comments for mock patches

  mvanniasingheTT

  committed
  40ba018

  Copy full SHA for 40ba018

• update vllm eval docs and development docs

  tstescoTT

  committed
  ed15e1c

  Copy full SHA for ed15e1c

Commits on Oct 28, 2024

• update mock to work with latest vllm commit 82dbca6

  mvanniasingheTT

  committed
  f2b1f1b

  Copy full SHA for f2b1f1b

• remove tracing calls in mock model

  mvanniasingheTT

  committed
  65ff822

  Copy full SHA for 65ff822

• remove uneeded imports

  mvanniasingheTT

  committed
  16320de

  Copy full SHA for 16320de

• remove checkout specfic commit

  mvanniasingheTT

  committed
  57a102c

  Copy full SHA for 57a102c

• remove imports and comments

  mvanniasingheTT

  committed
  01ee140

  Copy full SHA for 01ee140

• cleanup

  mvanniasingheTT

  committed
  7a92a47

  Copy full SHA for 7a92a47

• remove fork branch for tstescoTT/lm-evaluation-harness, can now use main

  tstescoTT

  committed
  42bd5a6

  Copy full SHA for 42bd5a6

• remove fork branch for tstescoTT/lm-evaluation-harness, can now use main

  tstescoTT

  committed
  f2ac1cc

  Copy full SHA for f2ac1cc

• adding TT_VLLM_COMMIT_SHA_OR_TAG to Dockerfile

  tstescoTT

  committed
  b7a9406

  Copy full SHA for b7a9406

• uplift tt-metal and vllm commit SHA

  tstescoTT

  committed
  77cb4ba

  Copy full SHA for 77cb4ba

Commits on Oct 30, 2024

• adding TT_VLLM_COMMIT_DOCKER_TAG in vllm dev docker build instructions

  tstescoTT

  committed
  7de2b00

  Copy full SHA for 7de2b00

• adding comment header for ifeval

  tstescoTT

  committed
  f1ad0e1

  Copy full SHA for f1ad0e1

• fix comment

  tstescoTT

  committed
  cbbc80e

  Copy full SHA for cbbc80e

• update vllm commit sha

  tstescoTT

  committed
  dc27981

  Copy full SHA for dc27981

• adding logging

  mvanniasingheTT

  committed
  68ffdb4

  Copy full SHA for 68ffdb4

• adding additonal tools, removing symlink for tt-metal models

  tstescoTT

  committed
  43dff21

  Copy full SHA for 43dff21

Commits on Oct 31, 2024

• update Dockerfile version in docs

  tstescoTT

  committed
  213d7ef

  Copy full SHA for 213d7ef

• remove debug tools

  tstescoTT

  committed
  fb12460

  Copy full SHA for fb12460

• make logger point to same logger in vllm dir to that logger.info() works

  mvanniasingheTT

  committed
  bdd86a0

  Copy full SHA for bdd86a0

• update tpot logging - NOTE: mult by num scheduler steps

  mvanniasingheTT

  committed
  ce17c50

  Copy full SHA for ce17c50

Commits on Nov 1, 2024

• resovled merge conflcits

  mvanniasingheTT

  committed
  f46656e

  Copy full SHA for f46656e

• update to take num scheduler steps from engine args

  mvanniasingheTT

  committed
  b2d8393

  Copy full SHA for b2d8393

• Merge branch 'mvanniasinghe/logging' of https://github.com/tenstorrent/tt-inference-server into mvanniasinghe/logging

  mvanniasingheTT

  committed
  1b36a4e

  Copy full SHA for 1b36a4e

Commits on Nov 6, 2024

• resolve merge conflict

  mvanniasingheTT

  committed
  fd873ab

  Copy full SHA for fd873ab

• resolve merge conflict + add logger_utils.py

  mvanniasingheTT

  committed
  5a7305c

  Copy full SHA for 5a7305c

Commits on Nov 7, 2024

• added online logging

  mvanniasingheTT

  committed
  93aaf10

  Copy full SHA for 93aaf10

Commits on Nov 8, 2024

• update MQLLMEngine to use scheduler_config to get num steps

  mvanniasingheTT

  committed
  c9d1ec1

  Copy full SHA for c9d1ec1

• update variable name rot_idxs_tt

  mvanniasingheTT

  committed
  ceb0fea

  Copy full SHA for ceb0fea

Commits on Nov 12, 2024

• clean up imports

  mvanniasingheTT

  committed
  4c84da4

  Copy full SHA for 4c84da4

• move new init over to logging utils + comments

  mvanniasingheTT

  committed
  c21a854

  Copy full SHA for c21a854

• added jsonl logging - issue with online

  mvanniasingheTT

  committed
  4e150a2

  Copy full SHA for 4e150a2

• remove breakpoint

  mvanniasingheTT

  committed
  e3efb5b

  Copy full SHA for e3efb5b

• added line by line logging for offlien

  mvanniasingheTT

  committed
  092fbbf

  Copy full SHA for 092fbbf

• updated loggin util with jsonl logger

  mvanniasingheTT

  committed
  e09b428

  Copy full SHA for e09b428

Commits on Nov 13, 2024

• added fix for when there is no decode (only generate first token/prefill

  mvanniasingheTT

  committed
  ea29384

  Copy full SHA for ea29384

• fix for only decode + better logging online - batch size hard coded

  mvanniasingheTT

  committed
  1f9d587

  Copy full SHA for 1f9d587

Commits on Nov 14, 2024

• update to ignore .jsonl

  mvanniasingheTT

  committed
  f9395b4

  Copy full SHA for f9395b4

• delete data

  mvanniasingheTT

  committed
  70472e7

  Copy full SHA for 70472e7

• added .env.llama31_8b

  mvanniasingheTT

  committed
  9dbc918

  Copy full SHA for 9dbc918

• update to use variable batch size - remove hard code

  mvanniasingheTT

  committed
  75471e8

  Copy full SHA for 75471e8

• update logging utils with changes

  mvanniasingheTT

  committed
  49694c0

  Copy full SHA for 49694c0