Serve UAE embed model in OpenAI compatible server?

[bioshazard]

I was excited to see we can now serve multiple models from a single instance of the OpenAI compatible endpoint! I was excited to try to serve a dedicated embedding model so I could keep my embeddings consistent while swapping out for an arbitrary completion model.

How can I serve this model? I tried to convert it to gguf but got an error I'll share later. Maybe this is a llama CPP question...

https://huggingface.co/WhereIsAI/UAE-Large-V1

[bioshazard]

llama.cpp> python .\convert.py ..\UAE-Large-V1\       
Loading model file ..\UAE-Large-V1\model.safetensors
Traceback (most recent call last):
  File "C:\Users\joe\Documents\root\workspaces\app-dev\llama.cpp\convert.py", line 1279, in <module>
    main()
  File "C:\Users\joe\Documents\root\workspaces\app-dev\llama.cpp\convert.py", line 1218, in main
    params = Params.load(model_plus)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\joe\Documents\root\workspaces\app-dev\llama.cpp\convert.py", line 318, in load
    params = Params.loadHFTransformerJson(model_plus.model, hf_config_path)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\joe\Documents\root\workspaces\app-dev\llama.cpp\convert.py", line 257, in loadHFTransformerJson
    f_norm_eps        = config["rms_norm_eps"],
                        ~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'rms_norm_eps'

[bioshazard]

ok looks like bert models are outright not supported by llama.cpp: NotImplementedError: Architecture "BertModel" not supported!

So... what are y'all using for embeddings? You just pick a random 7b? Bert models top the MTEB chart.

[bioshazard]

I found ggerganov/llama.cpp#2872 which linked me to https://github.com/xyzhang626/embeddings.cpp

Maybe I am pitching embeddings.cpp bindings in this project at least until llama.cpp gets official bert support from that first thread.