Skip to content

Commit

Permalink
[ez] Remove fixed TODO comments in HF text_generation local inference…
Browse files Browse the repository at this point in the history
… model parser

This was fixed in lastmile-ai#410, but forgot to remove the TODO, doing it in this diff
  • Loading branch information
Rossdan Craig [email protected] authored and Victor Su-Ortiz committed Jan 2, 2024
1 parent 6f16678 commit 238a108
Showing 1 changed file with 0 additions and 5 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -251,11 +251,6 @@ async def run_inference(
not "stream" in completion_data or completion_data.get("stream") != False
)
if should_stream:
# TODO (rossdanlm): I noticed that some models are incohorent when used as a tokenizer for streaming
# mistralai/Mistral-7B-v0.1 is able to generate text no problem, but doesn't make sense when it tries to tokenize
# in these cases, I would use `gpt2`. I'm wondering if there's a heuristic
# we can use to determine if a model is applicable for being used as a tokenizer
# For now I can just default the line below to gpt2? Maybe we can also define it somehow in the aiconfig?
tokenizer : AutoTokenizer = AutoTokenizer.from_pretrained(model_name)
streamer = TextIteratorStreamer(tokenizer)
completion_data["streamer"] = streamer
Expand Down

0 comments on commit 238a108

Please sign in to comment.