Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Logprobs in MLC Batch Serving #82

Merged
Changes from 1 commit
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
ab47b41
Squashed commit for logprobs implementation.
zxybazh Jan 22, 2024
86f6fa1
fix None check
Jan 23, 2024
9a29650
Change detokenization to using token ids.
zxybazh Jan 25, 2024
012388d
Fix wrong usage of token ids. Remove logging.
zxybazh Jan 29, 2024
db31164
extend benchmarks for logprobs
Jan 26, 2024
be81755
fix test without logprobs
Jan 26, 2024
e8ec3fc
clean code
Jan 26, 2024
49187f5
black format engine_common.py
Jan 26, 2024
013ed5a
logprobs is strictly bool, top_logprobs is int
Jan 26, 2024
79ec413
refactor logprob info collection to not reduce performance
Jan 28, 2024
fca1a6f
quick fix for check
Jan 29, 2024
675b631
review fix
Jan 29, 2024
18f80fa
fix list index out of range
Jan 29, 2024
29ea525
rollback after rebase
Jan 29, 2024
aa99322
test
Jan 29, 2024
8fa785e
Merge pull request #7 from Deelvin/vc/benchmark
Jan 29, 2024
d57b197
Squashed commit for logprobs implementation.
zxybazh Jan 22, 2024
7995c84
fix None check
Jan 23, 2024
ae3fc5b
Change detokenization to using token ids.
zxybazh Jan 25, 2024
0cb036f
Fix wrong usage of token ids. Remove logging.
zxybazh Jan 29, 2024
ed51e7d
extend benchmarks for logprobs
Jan 26, 2024
ff17ae2
fix test without logprobs
Jan 26, 2024
f5e4339
clean code
Jan 26, 2024
a3f6e8b
black format engine_common.py
Jan 26, 2024
c54a410
logprobs is strictly bool, top_logprobs is int
Jan 26, 2024
379d991
refactor logprob info collection to not reduce performance
Jan 28, 2024
58bac8f
quick fix for check
Jan 29, 2024
7de8d88
review fix
Jan 29, 2024
661fa18
fix list index out of range
Jan 29, 2024
6662a65
rollback after rebase
Jan 29, 2024
970d7f8
test
Jan 29, 2024
c58d69c
small fix
Jan 30, 2024
ebae200
rename for the sake of clarity
Jan 30, 2024
b2863d5
some fixes with cpu-gpu tensor copying
Jan 30, 2024
57b3a35
refactor logprob pass to calculate
Jan 30, 2024
4e29403
remove excess deps for token detokenization
Jan 30, 2024
a9157b9
small clean
Jan 30, 2024
39efb61
small clean
Jan 31, 2024
601e68d
return None instead of list of Nones
Jan 31, 2024
4f9241b
resolve conflicts
Jan 31, 2024
7ec21a7
fix mypy
Jan 31, 2024
7aa60ed
Merge pull request #8 from Deelvin/vc/perf
Jan 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
clean code
Valery Chernov committed Jan 29, 2024
commit e8ec3fc747a047a10249b51e8c42d3052f1825ca
8 changes: 4 additions & 4 deletions serve/mlc_serve/engine/engine_common.py
Original file line number Diff line number Diff line change
@@ -138,8 +138,8 @@ def detokenize_incrementally(


def logprob_detokenize(
tokenizer: TokenizerP,
logprob_info: Optional[RawLogprobsInfo],
tokenizer: TokenizerP,
logprob_info: Optional[RawLogprobsInfo],
) -> Optional[LogprobsContent]:
"""Detokenize tokens from RawLogprobInfo and convert the latter to LogprobContent"""
if logprob_info is None:
@@ -176,8 +176,8 @@ def logprob_detokenize(


def logprobs_detokenize(
tokenizer: TokenizerP,
logprob_info: List[Optional[RawLogprobsInfo]],
tokenizer: TokenizerP,
logprob_info: List[Optional[RawLogprobsInfo]],
) -> Optional[List[Optional[LogprobsContent]]]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you do something about this function. It's return type is too messy.

if logprob_info is None:
return None
2 changes: 1 addition & 1 deletion serve/mlc_serve/openai_logprob_protocol.py
Original file line number Diff line number Diff line change
@@ -25,4 +25,4 @@ class Logprobs(BaseModel):
See details in https://platform.openai.com/docs/api-reference/chat/object#chat-create-logprobs
"""

content: List[LogprobsContent]
content: List[LogprobsContent]