Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

负载均衡覆盖model_redirect吗? #90

Open
denniszlei opened this issue Dec 26, 2024 · 0 comments
Open

负载均衡覆盖model_redirect吗? #90

denniszlei opened this issue Dec 26, 2024 · 0 comments

Comments

@denniszlei
Copy link

请教一下,看到wiki里的说明:如果负载均衡为random,则服务选择: 如果同一个模型包含多个服务实例,则会通过随机策略选择一个服务实例。,这里面的多个实例,是否检测model_redeirect设置?比如下面的config,客户端请求gpt-4o-mini的时候,会服从random的负载均衡设置吗?

... ...
        {
          "models": [
            "cursor/gpt-4o",
            "cursor/gpt-4o-mini",
            "cursor/o1-mini"
          ],
          "enabled": true,
          "credentials": {
            "api_key": "AAA"
          },
          "model_redirect": {
            "gpt-4o": "cursor/gpt-4o",
            "gpt-4o-mini": "cursor/gpt-4o-mini",
            "gpt-o1-mini": "cursor/o1-mini"
          },
          "server_url":"http://1.1.1.1:18085/v1"
        },
        {
          "models": [
            "gpt-4o-mini"
          ],
          "enabled": true,
          "credentials": {
            "api_key": "BBB"
          },
          "server_url":"http://2.2.2.2:18081/v1"
        }
... ...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant