Model cannot terminate itself when running single GGUF/GGML model

[guoqingbao]

Describe the bug

To reproduce this bug, follow these steps:

1. Download the model Phi-3.5-mini-instruct-Q4_K_M.gguf to a folder (e.g., /Users/Downloads/).

2. Run the following command to build and run the project:

cargo build --release --features metal
./target/release/mistralrs-server -i --throughput gguf -m /Users/Downloads/ -f Phi-3.5-mini-instruct-Q4_K_M.gguf

3.Chat with the model, and it will continuously produce text without terminating.

Temporary Fix:

A temporary fix involves parsing the EOS token (in paths.rs) from the literal chat template within the GGUF file.

    let mut template: ChatTemplate = match chat_template_ovrd {
        Some(chat_template) => {
            // In this case the override chat template is being used. The user must add the bos/eos/unk toks themselves.
            info!("Using literal chat template.");
            let mut template = ChatTemplate::default();
            if chat_template.find("<|end|>").is_some() {
                template.eos_token = Some(BeginEndUnkTok(Either::Left("<|end|>".to_string())));
            } else if chat_template.find("<|endoftext|>").is_some() {
                template.eos_token =
                    Some(BeginEndUnkTok(Either::Left("<|endoftext|>".to_string())));
            }
            template.chat_template = Some(ChatTemplateValue(Either::Left(chat_template)));
            template
        }
        None => serde_json::from_str(&template_content.as_ref().unwrap().clone()).unwrap(),
    };

Latest commit or version

Latest version: d28ddf9