JSON schema creation on llama cpp failing

[motaatmo]

Describe the issue as clearly as possible:

I'm trying to generate using a JSON schema on llama.cpp. This throws lots of errors and creates wrong output.

Steps/code to reproduce the bug:

import outlines
import json
import os
import sys

def raw_outlines_test():
    """
    Just a test

    """
    schema = """{"properties": {"name": {"title": "Name", "type": "string",
    "maxLength": 100}},
    "required": ["name"], "title": "person", "type": "object"}
    """
    import sys
    json.loads(schema)
    sys.stderr.write("XXXXXXXXXXX  JSON schema successfully created  XXXXXXXXXXXXXX\n")
    llm = outlines.models.llamacpp(
        os.path.join(
            os.environ["TRANSFORMERS_CACHE"],
            "Nous-Capybara-34B-GGUF/nous-capybara-34b.Q4_0.gguf"),
        model_kwargs = {"n_gpu_layers": 15000, "n_ctx": 5 * 1024})
    sys.stderr.write("XXXXXXXXXX Model loaded  XXXXXXXXXXXXXXXXXXXXXX\n")
    generator = outlines.generate.json(llm, schema)
    sys.stderr.write("XXXXXXXXXXXX Generator created XXXXXXXXXXXXXXX\n")
    text = generator("What is the name of Michael Jackson?")
    sys.stderr.write("XXXXXXXXXXXXX Result: XXXXXXXXXXXXXXXX\n")
    sys.stderr.write(str(text))

Expected result:

{
	"name": "Michael Jackson"
}

Error message:

XXXXXXXXXXX  JSON schema successfully created  XXXXXXXXXXXXXX
ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA A100-SXM4-40GB, compute capability 8.0, VMM: yes
llama_model_loader: loaded meta data with 22 key-value pairs and 543 tensors from /misc/cache/huggingface/models/Nous-Capybara-34B-GGUF/nous-capybara-34b.Q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
llama_model_loader: - kv   1:                               general.name str              = nousresearch_nous-capybara-34b
llama_model_loader: - kv   2:                       llama.context_length u32              = 200000
llama_model_loader: - kv   3:                     llama.embedding_length u32              = 7168
llama_model_loader: - kv   4:                          llama.block_count u32              = 60
llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 20480
llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128
llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 56
llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 8
llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 5000000.000000
llama_model_loader: - kv  11:                          general.file_type u32              = 2
llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,64000]   = ["<unk>", "<|startoftext|>", "<|endof...
llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,64000]   = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,64000]   = [2, 3, 3, 3, 3, 3, 1, 1, 1, 3, 3, 3, ...
llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1
llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2
llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  21:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q4_0:  421 tensors
llama_model_loader: - type q6_K:    1 tensors
llm_load_vocab: mismatch in special tokens definition ( 498/64000 vs 267/64000 ).
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = SPM
llm_load_print_meta: n_vocab          = 64000
llm_load_print_meta: n_merges         = 0
llm_load_print_meta: n_ctx_train      = 200000
llm_load_print_meta: n_embd           = 7168
llm_load_print_meta: n_head           = 56
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_layer          = 60
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 7
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: n_ff             = 20480
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 5000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx  = 200000
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: model type       = 30B
llm_load_print_meta: model ftype      = Q4_0
llm_load_print_meta: model params     = 34.39 B
llm_load_print_meta: model size       = 18.13 GiB (4.53 BPW) 
llm_load_print_meta: general.name     = nousresearch_nous-capybara-34b
llm_load_print_meta: BOS token        = 1 '<|startoftext|>'
llm_load_print_meta: EOS token        = 2 '<|endoftext|>'
llm_load_print_meta: UNK token        = 0 '<unk>'
llm_load_print_meta: PAD token        = 0 '<unk>'
llm_load_print_meta: LF token         = 315 '<0x0A>'
llm_load_tensors: ggml ctx size =    0.41 MiB
llm_load_tensors: offloading 60 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 61/61 layers to GPU
llm_load_tensors:        CPU buffer size =   246.09 MiB
llm_load_tensors:      CUDA0 buffer size = 18317.20 MiB
...................................................................................................
Failed to convert token (b'\x80'): 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Failed to convert token (b'\x81'): 'utf-8' codec can't decode byte 0x81 in position 0: invalid start byte
Failed to convert token (b'\x82'): 'utf-8' codec can't decode byte 0x82 in position 0: invalid start byte
Failed to convert token (b'\x83'): 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
Failed to convert token (b'\x84'): 'utf-8' codec can't decode byte 0x84 in position 0: invalid start byte
Failed to convert token (b'\x85'): 'utf-8' codec can't decode byte 0x85 in position 0: invalid start byte
Failed to convert token (b'\x86'): 'utf-8' codec can't decode byte 0x86 in position 0: invalid start byte
Failed to convert token (b'\x87'): 'utf-8' codec can't decode byte 0x87 in position 0: invalid start byte
Failed to convert token (b'\x88'): 'utf-8' codec can't decode byte 0x88 in position 0: invalid start byte
Failed to convert token (b'\x89'): 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte
Failed to convert token (b'\x8a'): 'utf-8' codec can't decode byte 0x8a in position 0: invalid start byte
Failed to convert token (b'\x8b'): 'utf-8' codec can't decode byte 0x8b in position 0: invalid start byte
Failed to convert token (b'\x8c'): 'utf-8' codec can't decode byte 0x8c in position 0: invalid start byte
Failed to convert token (b'\x8d'): 'utf-8' codec can't decode byte 0x8d in position 0: invalid start byte
Failed to convert token (b'\x8e'): 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
Failed to convert token (b'\x8f'): 'utf-8' codec can't decode byte 0x8f in position 0: invalid start byte
Failed to convert token (b'\x90'): 'utf-8' codec can't decode byte 0x90 in position 0: invalid start byte
Failed to convert token (b'\x91'): 'utf-8' codec can't decode byte 0x91 in position 0: invalid start byte
Failed to convert token (b'\x92'): 'utf-8' codec can't decode byte 0x92 in position 0: invalid start byte
Failed to convert token (b'\x93'): 'utf-8' codec can't decode byte 0x93 in position 0: invalid start byte
Failed to convert token (b'\x94'): 'utf-8' codec can't decode byte 0x94 in position 0: invalid start byte
Failed to convert token (b'\x95'): 'utf-8' codec can't decode byte 0x95 in position 0: invalid start byte
Failed to convert token (b'\x96'): 'utf-8' codec can't decode byte 0x96 in position 0: invalid start byte
Failed to convert token (b'\x97'): 'utf-8' codec can't decode byte 0x97 in position 0: invalid start byte
Failed to convert token (b'\x98'): 'utf-8' codec can't decode byte 0x98 in position 0: invalid start byte
Failed to convert token (b'\x99'): 'utf-8' codec can't decode byte 0x99 in position 0: invalid start byte
Failed to convert token (b'\x9a'): 'utf-8' codec can't decode byte 0x9a in position 0: invalid start byte
Failed to convert token (b'\x9b'): 'utf-8' codec can't decode byte 0x9b in position 0: invalid start byte
Failed to convert token (b'\x9c'): 'utf-8' codec can't decode byte 0x9c in position 0: invalid start byte
Failed to convert token (b'\x9d'): 'utf-8' codec can't decode byte 0x9d in position 0: invalid start byte
Failed to convert token (b'\x9e'): 'utf-8' codec can't decode byte 0x9e in position 0: invalid start byte
Failed to convert token (b'\x9f'): 'utf-8' codec can't decode byte 0x9f in position 0: invalid start byte
Failed to convert token (b'\xa0'): 'utf-8' codec can't decode byte 0xa0 in position 0: invalid start byte
Failed to convert token (b'\xa1'): 'utf-8' codec can't decode byte 0xa1 in position 0: invalid start byte
Failed to convert token (b'\xa2'): 'utf-8' codec can't decode byte 0xa2 in position 0: invalid start byte
Failed to convert token (b'\xa3'): 'utf-8' codec can't decode byte 0xa3 in position 0: invalid start byte
Failed to convert token (b'\xa4'): 'utf-8' codec can't decode byte 0xa4 in position 0: invalid start byte
Failed to convert token (b'\xa5'): 'utf-8' codec can't decode byte 0xa5 in position 0: invalid start byte
Failed to convert token (b'\xa6'): 'utf-8' codec can't decode byte 0xa6 in position 0: invalid start byte
Failed to convert token (b'\xa7'): 'utf-8' codec can't decode byte 0xa7 in position 0: invalid start byte
Failed to convert token (b'\xa8'): 'utf-8' codec can't decode byte 0xa8 in position 0: invalid start byte
Failed to convert token (b'\xa9'): 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte
Failed to convert token (b'\xaa'): 'utf-8' codec can't decode byte 0xaa in position 0: invalid start byte
Failed to convert token (b'\xab'): 'utf-8' codec can't decode byte 0xab in position 0: invalid start byte
Failed to convert token (b'\xac'): 'utf-8' codec can't decode byte 0xac in position 0: invalid start byte
Failed to convert token (b'\xad'): 'utf-8' codec can't decode byte 0xad in position 0: invalid start byte
Failed to convert token (b'\xae'): 'utf-8' codec can't decode byte 0xae in position 0: invalid start byte
Failed to convert token (b'\xaf'): 'utf-8' codec can't decode byte 0xaf in position 0: invalid start byte
Failed to convert token (b'\xb0'): 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte
Failed to convert token (b'\xb1'): 'utf-8' codec can't decode byte 0xb1 in position 0: invalid start byte
Failed to convert token (b'\xb2'): 'utf-8' codec can't decode byte 0xb2 in position 0: invalid start byte
Failed to convert token (b'\xb3'): 'utf-8' codec can't decode byte 0xb3 in position 0: invalid start byte
Failed to convert token (b'\xb4'): 'utf-8' codec can't decode byte 0xb4 in position 0: invalid start byte
Failed to convert token (b'\xb5'): 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte
Failed to convert token (b'\xb6'): 'utf-8' codec can't decode byte 0xb6 in position 0: invalid start byte
Failed to convert token (b'\xb7'): 'utf-8' codec can't decode byte 0xb7 in position 0: invalid start byte
Failed to convert token (b'\xb8'): 'utf-8' codec can't decode byte 0xb8 in position 0: invalid start byte
Failed to convert token (b'\xb9'): 'utf-8' codec can't decode byte 0xb9 in position 0: invalid start byte
Failed to convert token (b'\xba'): 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte
Failed to convert token (b'\xbb'): 'utf-8' codec can't decode byte 0xbb in position 0: invalid start byte
Failed to convert token (b'\xbc'): 'utf-8' codec can't decode byte 0xbc in position 0: invalid start byte
Failed to convert token (b'\xbd'): 'utf-8' codec can't decode byte 0xbd in position 0: invalid start byte
Failed to convert token (b'\xbe'): 'utf-8' codec can't decode byte 0xbe in position 0: invalid start byte
Failed to convert token (b'\xbf'): 'utf-8' codec can't decode byte 0xbf in position 0: invalid start byte
Failed to convert token (b'\xc0'): 'utf-8' codec can't decode byte 0xc0 in position 0: invalid start byte
Failed to convert token (b'\xc1'): 'utf-8' codec can't decode byte 0xc1 in position 0: invalid start byte
Failed to convert token (b'\xc2'): 'utf-8' codec can't decode byte 0xc2 in position 0: unexpected end of data
Failed to convert token (b'\xc3'): 'utf-8' codec can't decode byte 0xc3 in position 0: unexpected end of data
Failed to convert token (b'\xc4'): 'utf-8' codec can't decode byte 0xc4 in position 0: unexpected end of data
Failed to convert token (b'\xc5'): 'utf-8' codec can't decode byte 0xc5 in position 0: unexpected end of data
Failed to convert token (b'\xc6'): 'utf-8' codec can't decode byte 0xc6 in position 0: unexpected end of data
Failed to convert token (b'\xc7'): 'utf-8' codec can't decode byte 0xc7 in position 0: unexpected end of data
Failed to convert token (b'\xc8'): 'utf-8' codec can't decode byte 0xc8 in position 0: unexpected end of data
Failed to convert token (b'\xc9'): 'utf-8' codec can't decode byte 0xc9 in position 0: unexpected end of data
Failed to convert token (b'\xca'): 'utf-8' codec can't decode byte 0xca in position 0: unexpected end of data
Failed to convert token (b'\xcb'): 'utf-8' codec can't decode byte 0xcb in position 0: unexpected end of data
llama_new_context_with_model: n_ctx      = 5120
llama_new_context_with_model: freq_base  = 5000000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init:      CUDA0 KV buffer size =  1200.00 MiB
llama_new_context_with_model: KV self size  = 1200.00 MiB, K (f16):  600.00 MiB, V (f16):  600.00 MiB
llama_new_context_with_model:  CUDA_Host input buffer size   =    24.02 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =   673.20 MiB
llama_new_context_with_model:  CUDA_Host compute buffer size =    15.40 MiB
llama_new_context_with_model: graph splits (measure): 3
XXXXXXXXXX Model loaded  XXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXX Generator created XXXXXXXXXXXXXXX
XXXXXXXXXXXXX Result: XXXXXXXXXXXXXXXX
Failed to convert token (b'\xcc'): 'utf-8' codec can't decode byte 0xcc in position 0: unexpected end of data
Failed to convert token (b'\xcd'): 'utf-8' codec can't decode byte 0xcd in position 0: unexpected end of data
Failed to convert token (b'\xce'): 'utf-8' codec can't decode byte 0xce in position 0: unexpected end of data
Failed to convert token (b'\xcf'): 'utf-8' codec can't decode byte 0xcf in position 0: unexpected end of data
Failed to convert token (b'\xd0'): 'utf-8' codec can't decode byte 0xd0 in position 0: unexpected end of data
Failed to convert token (b'\xd1'): 'utf-8' codec can't decode byte 0xd1 in position 0: unexpected end of data
Failed to convert token (b'\xd2'): 'utf-8' codec can't decode byte 0xd2 in position 0: unexpected end of data
Failed to convert token (b'\xd3'): 'utf-8' codec can't decode byte 0xd3 in position 0: unexpected end of data
Failed to convert token (b'\xd4'): 'utf-8' codec can't decode byte 0xd4 in position 0: unexpected end of data
Failed to convert token (b'\xd5'): 'utf-8' codec can't decode byte 0xd5 in position 0: unexpected end of data
Failed to convert token (b'\xd6'): 'utf-8' codec can't decode byte 0xd6 in position 0: unexpected end of data
Failed to convert token (b'\xd7'): 'utf-8' codec can't decode byte 0xd7 in position 0: unexpected end of data
Failed to convert token (b'\xd8'): 'utf-8' codec can't decode byte 0xd8 in position 0: unexpected end of data
Failed to convert token (b'\xd9'): 'utf-8' codec can't decode byte 0xd9 in position 0: unexpected end of data
Failed to convert token (b'\xda'): 'utf-8' codec can't decode byte 0xda in position 0: unexpected end of data
Failed to convert token (b'\xdb'): 'utf-8' codec can't decode byte 0xdb in position 0: unexpected end of data
Failed to convert token (b'\xdc'): 'utf-8' codec can't decode byte 0xdc in position 0: unexpected end of data
Failed to convert token (b'\xdd'): 'utf-8' codec can't decode byte 0xdd in position 0: unexpected end of data
Failed to convert token (b'\xde'): 'utf-8' codec can't decode byte 0xde in position 0: unexpected end of data
Failed to convert token (b'\xdf'): 'utf-8' codec can't decode byte 0xdf in position 0: unexpected end of data
Failed to convert token (b'\xe0'): 'utf-8' codec can't decode byte 0xe0 in position 0: unexpected end of data
Failed to convert token (b'\xe1'): 'utf-8' codec can't decode byte 0xe1 in position 0: unexpected end of data
Failed to convert token (b'\xe2'): 'utf-8' codec can't decode byte 0xe2 in position 0: unexpected end of data
Failed to convert token (b'\xe3'): 'utf-8' codec can't decode byte 0xe3 in position 0: unexpected end of data
Failed to convert token (b'\xe4'): 'utf-8' codec can't decode byte 0xe4 in position 0: unexpected end of data
Failed to convert token (b'\xe5'): 'utf-8' codec can't decode byte 0xe5 in position 0: unexpected end of data
Failed to convert token (b'\xe6'): 'utf-8' codec can't decode byte 0xe6 in position 0: unexpected end of data
Failed to convert token (b'\xe7'): 'utf-8' codec can't decode byte 0xe7 in position 0: unexpected end of data
Failed to convert token (b'\xe8'): 'utf-8' codec can't decode byte 0xe8 in position 0: unexpected end of data
Failed to convert token (b'\xe9'): 'utf-8' codec can't decode byte 0xe9 in position 0: unexpected end of data
Failed to convert token (b'\xea'): 'utf-8' codec can't decode byte 0xea in position 0: unexpected end of data
Failed to convert token (b'\xeb'): 'utf-8' codec can't decode byte 0xeb in position 0: unexpected end of data
Failed to convert token (b'\xec'): 'utf-8' codec can't decode byte 0xec in position 0: unexpected end of data
Failed to convert token (b'\xed'): 'utf-8' codec can't decode byte 0xed in position 0: unexpected end of data
Failed to convert token (b'\xee'): 'utf-8' codec can't decode byte 0xee in position 0: unexpected end of data
Failed to convert token (b'\xef'): 'utf-8' codec can't decode byte 0xef in position 0: unexpected end of data
Failed to convert token (b'\xf0'): 'utf-8' codec can't decode byte 0xf0 in position 0: unexpected end of data
Failed to convert token (b'\xf1'): 'utf-8' codec can't decode byte 0xf1 in position 0: unexpected end of data
Failed to convert token (b'\xf2'): 'utf-8' codec can't decode byte 0xf2 in position 0: unexpected end of data
Failed to convert token (b'\xf3'): 'utf-8' codec can't decode byte 0xf3 in position 0: unexpected end of data
Failed to convert token (b'\xf4'): 'utf-8' codec can't decode byte 0xf4 in position 0: unexpected end of data
Failed to convert token (b'\xf5'): 'utf-8' codec can't decode byte 0xf5 in position 0: invalid start byte
Failed to convert token (b'\xf6'): 'utf-8' codec can't decode byte 0xf6 in position 0: invalid start byte
Failed to convert token (b'\xf7'): 'utf-8' codec can't decode byte 0xf7 in position 0: invalid start byte
Failed to convert token (b'\xf8'): 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
Failed to convert token (b'\xf9'): 'utf-8' codec can't decode byte 0xf9 in position 0: invalid start byte
Failed to convert token (b'\xfa'): 'utf-8' codec can't decode byte 0xfa in position 0: invalid start byte
Failed to convert token (b'\xfb'): 'utf-8' codec can't decode byte 0xfb in position 0: invalid start byte
Failed to convert token (b'\xfc'): 'utf-8' codec can't decode byte 0xfc in position 0: invalid start byte
Failed to convert token (b'\xfd'): 'utf-8' codec can't decode byte 0xfd in position 0: invalid start byte
Failed to convert token (b'\xfe'): 'utf-8' codec can't decode byte 0xfe in position 0: invalid start byte
Failed to convert token (b'\xff'): 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
{'name': "      [TYPE                     : String     <'aecollertlqpangstrom'      )-> int};</ pb>q_ -> reg->"}

Outlines/Python version information:

Version information

``` 0.0.25

Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]

</details>


### Context for the issue:

Outlines is currently not usable for me as I'm relying on a JSON schema functionality.

[lapp0]

Please share your llama_cpp_python version.

If it's not the latest, please do still share, but also upgrade and try again, and let me know your results.

[motaatmo]

It's 0.2.36, so I'll upgrade and retry,
greetings!

[motaatmo]

The upgrade didn't change much. The error messages seem unchanged; the returned value is, most of the time, {'name': ' '}, but scarier returns of strange memory contents ({'name': '}</s><commit_msg>Update Italian translation<commit_after>{</<s>{name> configuration values frompt-</'}) happen as well. I once even saw the string "passwd" in the returned data...

[lapp0]

Thanks for trying, sorry it's not still working as expected. Will investigate.

[motaatmo]

So you can reproduce the error? That gives me at least some peace of mind (if the problem doesn't seem to be caused by my setup...) Andrew Lapp ***@***.***> schrieb am Do., 8. Feb. 2024, 04:29:

…

Thanks for trying, sorry it's not still working as expected. Will investigate. — Reply to this email directly, view it on GitHub <#611 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACO5DS54SLJMHDPBX33FIB3YSRBBBAVCNFSM6AAAAABC2JJ2JCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMZTGMYDMOJXGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[thepatattorney]

Dear all,
It is my first message ever on GitHub. Pardon me if I am doing something wrong.
Yesterday, I installed the last version of outlines and python-llama-cpp and I have a similar series of error messages "failed to convert token", whatever the prompt or the gguf model loaded. I am using CPU only.

[rlouf]

There is a PR open that should solve most issues with llama.cpp (#556), I will review it as quickly as I can

[lapp0]

@motaatmo my test suggests your issue is resolved by #556

You may try it early via pip install git+https://github.com/dtiarks/outlines@llama-logits-processor && pip install llama-cpp-python==0.2.37

[motaatmo]

Yes, it works! No more errors, however one minor thing: the returned JSON is using single rather than double quotes, which is, as far as I know, not JSON-compliant

[lapp0]

Yes, it works! No more errors, however one minor thing: the returned JSON is using single rather than double quotes, which is, as far as I know, not JSON-compliant

This shouldn't be possible with Outlines and is a bug. Could you share the script you're running in a new issue?