New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Configuration setting to pass parameters to tokenizer while encoding and decoding #522

Open

krishnanpooja opened this issue Aug 29, 2024 · 0 comments

krishnanpooja commented Aug 29, 2024

We would like to have configuration options to pass parameters while encoding and decoding using Huggingface tokenizer. Ex:-
in

DeepSpeed-MII/mii/batching/ragged_batching.py

Line 630 in 2d5a2e7

input_tokens = self.tokenizer.encode(input)

, we need option to set truncation based on the max_length

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment