Code

Enable Logprobs in MLC Batch Serving#82

Code

Merged

masahi merged 42 commits intooctoml:batch-servingoctoml/mlc-llm:batch-servingfrom zxybazh:feature/2023-11-22/enable-mlc-server-logprobszxybazh/mlc-llm:feature/2023-11-22/enable-mlc-server-logprobsCopy head branch name to clipboardJan 31, 2024

+376-53

Commits

Commits on Jan 25, 2024

• Squashed commit for logprobs implementation.

  Show description for ab47b41
  

  3 peoplecommitted
  ab47b41

  Copy full SHA for ab47b41

• fix None check

  

  Valery Chernov

  authored and

  zxybazh

  committed
  86f6fa1

  Copy full SHA for 86f6fa1

• Change detokenization to using token ids.

  zxybazh

  committed
  9a29650

  Copy full SHA for 9a29650

Commits on Jan 29, 2024

• Fix wrong usage of token ids. Remove logging.

  zxybazh

  committed
  012388d

  Copy full SHA for 012388d

• extend benchmarks for logprobs

  

  Valery Chernov

  committed
  db31164

  Copy full SHA for db31164

• fix test without logprobs

  

  Valery Chernov

  committed
  be81755

  Copy full SHA for be81755

• clean code

  

  Valery Chernov

  committed
  e8ec3fc

  Copy full SHA for e8ec3fc

• black format engine_common.py

  

  Valery Chernov

  committed
  49187f5

  Copy full SHA for 49187f5

• logprobs is strictly bool, top_logprobs is int

  

  Valery Chernov

  committed
  013ed5a

  Copy full SHA for 013ed5a

• refactor logprob info collection to not reduce performance

  

  Valery Chernov

  committed
  79ec413

  Copy full SHA for 79ec413

• quick fix for check

  

  Valery Chernov

  committed
  fca1a6f

  Copy full SHA for fca1a6f

• review fix

  

  Valery Chernov

  committed
  675b631

  Copy full SHA for 675b631

• fix list index out of range

  

  Valery Chernov

  committed
  18f80fa

  Copy full SHA for 18f80fa

• rollback after rebase

  

  Valery Chernov

  committed
  29ea525

  Copy full SHA for 29ea525

• test

  

  Valery Chernov

  committed
  aa99322

  Copy full SHA for aa99322

• Merge pull request #7 from Deelvin/vc/benchmark

  Show description for 8fa785e
  

  Valery Chernov

  authored
  8fa785e

  Copy full SHA for 8fa785e

Commits on Jan 31, 2024

• Squashed commit for logprobs implementation.

  Show description for d57b197
  

  3 peopleauthored and

  Valery Chernov

  committed
  d57b197

  Copy full SHA for d57b197

• fix None check

  

  Valery Chernov

  committed
  7995c84

  Copy full SHA for 7995c84

• Change detokenization to using token ids.

  

  zxybazh

  authored and

  Valery Chernov

  committed
  ae3fc5b

  Copy full SHA for ae3fc5b

• Fix wrong usage of token ids. Remove logging.

  

  zxybazh

  authored and

  Valery Chernov

  committed
  0cb036f

  Copy full SHA for 0cb036f

• extend benchmarks for logprobs

  

  Valery Chernov

  committed
  ed51e7d

  Copy full SHA for ed51e7d

• fix test without logprobs

  

  Valery Chernov

  committed
  ff17ae2

  Copy full SHA for ff17ae2

• clean code

  

  Valery Chernov

  committed
  f5e4339

  Copy full SHA for f5e4339

• black format engine_common.py

  

  Valery Chernov

  committed
  a3f6e8b

  Copy full SHA for a3f6e8b

• logprobs is strictly bool, top_logprobs is int

  

  Valery Chernov

  committed
  c54a410

  Copy full SHA for c54a410

• refactor logprob info collection to not reduce performance

  

  Valery Chernov

  committed
  379d991

  Copy full SHA for 379d991

• quick fix for check

  

  Valery Chernov

  committed
  58bac8f

  Copy full SHA for 58bac8f

• review fix

  

  Valery Chernov

  committed
  7de8d88

  Copy full SHA for 7de8d88

• fix list index out of range

  

  Valery Chernov

  committed
  661fa18

  Copy full SHA for 661fa18

• rollback after rebase

  

  Valery Chernov

  committed
  6662a65

  Copy full SHA for 6662a65

• test

  

  Valery Chernov

  committed
  970d7f8

  Copy full SHA for 970d7f8

• small fix

  

  Valery Chernov

  committed
  c58d69c

  Copy full SHA for c58d69c

• rename for the sake of clarity

  

  Valery Chernov

  committed
  ebae200

  Copy full SHA for ebae200

• some fixes with cpu-gpu tensor copying

  

  Valery Chernov

  committed
  b2863d5

  Copy full SHA for b2863d5

• refactor logprob pass to calculate

  

  Valery Chernov

  committed
  57b3a35

  Copy full SHA for 57b3a35

• remove excess deps for token detokenization

  

  Valery Chernov

  committed
  4e29403

  Copy full SHA for 4e29403

• small clean

  

  Valery Chernov

  committed
  a9157b9

  Copy full SHA for a9157b9

• small clean

  

  Valery Chernov

  committed
  39efb61

  Copy full SHA for 39efb61

• return None instead of list of Nones

  

  Valery Chernov

  committed
  601e68d

  Copy full SHA for 601e68d

• resolve conflicts

  

  Valery Chernov

  committed
  4f9241b

  Copy full SHA for 4f9241b

• fix mypy

  

  Valery Chernov

  committed
  7ec21a7

  Copy full SHA for 7ec21a7

• Merge pull request #8 from Deelvin/vc/perf

  Show description for 7aa60ed
  

  Valery Chernov

  authored
  7aa60ed

  Copy full SHA for 7aa60ed