how to calculator model input/output tokens length with java

[gabohouhou]

with python，here is an example :
`from transformers import AutoTokenizer

input_text_list = ["who are you?","how old are you","color see see"]
TOKENIZER = AutoTokenizer.from_pretrained("./model_params")
model_inputs = TOKENIZER(input_text_list,
# truncation="longest_first",
padding=True, return_tensors="pt")

print('model_inputs: ', model_inputs)

now i need a java sdk to implement calculator model_input tokens length.

i learn djl tokenization, but i can not find how to load voc、tokenization.json from local disk。

[frankfliu]

Here is example:

https://github.com/deepjavalibrary/djl/blob/master/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java#L114-L115

[gabohouhou]

thank you very much！