[Bug]: [2.5.x] [etcd] etcdserver: invalid auth token

[tmnhy]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.5.3
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): 2.5.3
- OS(Ubuntu or CentOS): official docker image: milvusdb/milvus:v2.5.3
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Create new database and database admin role and user with pymilvus. When I try to use a new admin user I get an error

025-01-20 07:42:24,355 [ERROR][handler]: RPC error: [list_collections], <MilvusException: (code=65535, message=etcdserver: invalid auth token)>, <Time:{'RPC start': '2025-01-20 07:42:24.351568', 'RPC error': '2025-01-20 07:42:24.355179'}> (decorators.py:140)

Code for reproduction:

from pymilvus import MilvusClient

URI = 'https://...'
ROOT_TOKEN = 'root:...'

DB_NAME = 'test'
ADMIN_NAME = 'admin'
ADMIN_PASSWORD = 'password'
ADMIN_TOKEN = f'{ADMIN_NAME}:{ADMIN_PASSWORD}'
ADMIN_ROLE_NAME = 'admin_role'

COLLECTION_NAME = 'test_collection'

# Create database
#
client = MilvusClient(uri=URI, db_name='default', token=ROOT_TOKEN)
client.create_database(DB_NAME)
client.close()

# Create admin role and user on test DB
client = MilvusClient(uri=URI, db_name=DB_NAME, token=ROOT_TOKEN)
client.create_role(ADMIN_ROLE_NAME)
client.grant_privilege_v2(role_name=ADMIN_ROLE_NAME, privilege="DatabaseAdmin", collection_name="*", db_name=DB_NAME)
client.create_user(ADMIN_NAME, ADMIN_PASSWORD)
client.grant_role(ADMIN_NAME, ADMIN_ROLE_NAME)
client.close()

client = MilvusClient(uri=URI, db_name=DB_NAME, token=ADMIN_TOKEN)
client.list_collections()
client.close()

# 025-01-20 07:42:24,355 [ERROR][handler]: RPC error: [list_collections], <MilvusException: (code=65535, message=etcdserver: invalid auth token)>, <Time:{'RPC start': '2025-01-20 07:42:24.351568', 'RPC error': '2025-01-20 07:42:24.355179'}> (decorators.py:140)

But at the same time:

client = MilvusClient(uri=URI, db_name=DB_NAME, token=ROOT_TOKEN)
print(client.describe_role(ADMIN_ROLE_NAME))
# {'role': 'admin_role', 'privileges': [{'object_type': 'Global', 'object_name': '*', 'db_name': 'test', 'role_name': 'admin_role', 'privilege': 'DatabaseAdmin', 'grantor_name': 'root'}]}
#

print(client.describe_user(ADMIN_NAME))
# {'user_name': 'user', 'roles': ('admin_role',)}

Expected Behavior

Normal user authorization.

Steps To Reproduce

The code is listed above.

Milvus Log

[2025/01/20 07:42:24.348 +00:00] [INFO] [proxy/impl.go:6275] ["connect received"] [traceID=a69c8aaa6ec019cd10ca3b67efd69501] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-20 07:42:24.346916"] [user=] [host=jupyterlab-3-test-jupyter-8c4686d88-7mc8v] [db=test]
[2025/01/20 07:42:24.348 +00:00] [INFO] [rootcoord/root_coord.go:993] ["received request to list databases"] [traceID=a69c8aaa6ec019cd10ca3b67efd69501] [msgID=0]
[2025/01/20 07:42:24.348 +00:00] [WARN] [rootcoord/list_db_task.go:56] ["get current user from context failed"] [traceID=a69c8aaa6ec019cd10ca3b67efd69501] [error="fail to get authorization from the md, authorization:[token]"]
[2025/01/20 07:42:24.348 +00:00] [INFO] [rootcoord/root_coord.go:1019] ["done to list databases"] [traceID=a69c8aaa6ec019cd10ca3b67efd69501] [msgID=0] ["num of databases"=2]
[2025/01/20 07:42:24.350 +00:00] [INFO] [connection/manager.go:106] ["client register"] [traceID=a69c8aaa6ec019cd10ca3b67efd69501] [sdk_type=Python] [sdk_version=2.5.3] [local_time="2025-01-20 07:42:24.346916"] [user=] [host=jupyterlab-3-test-jupyter-8c4686d88-7mc8v] [identifier=455438223101132802] [last_active_time=2025/01/20 07:42:24.350 +00:00]
{"level":"warn","ts":"2025-01-20T07:42:24.353Z","caller":"auth/store.go:1175","msg":"invalid auth token","token":"YWRtaW46YWRtaW4="}
[2025/01/20 07:42:24.353 +00:00] [WARN] [rootcoord/kv_catalog.go:567] ["get credential meta fail"] [traceID=f4f4f04e1d9c0b7a4b579a5d052d72a0] [key=root-coord/credential/users/admin] [error="etcdserver: invalid auth token"]
[2025/01/20 07:42:24.353 +00:00] [INFO] [rootcoord/root_coord.go:1317] ["failed to show collections"] [traceID=f4f4f04e1d9c0b7a4b579a5d052d72a0] [dbname=test] [ts=18446744073709551615] [error="etcdserver: invalid auth token"]
[2025/01/20 07:42:24.354 +00:00] [WARN] [proxy/task_scheduler.go:480] ["Failed to execute task: "] [traceID=f4f4f04e1d9c0b7a4b579a5d052d72a0] [error="etcdserver: invalid auth token"]
[2025/01/20 07:42:24.354 +00:00] [WARN] [proxy/impl.go:1232] ["ShowCollections failed to WaitToFinish"] [traceID=f4f4f04e1d9c0b7a4b579a5d052

Anything else?

I'm seeing strange things in the logs:

{"level":"warn","ts":"2025-01-20T07:42:24.353Z","caller":"auth/store.go:1175","msg":"invalid auth token","token":"YWRtaW46YWRtaW4="}

as "YWRtaW46YWRtaW4=" is "admin:admin" instead of "admin:password".

[xiaofan-luan]

@tmnhy
I guess this is not milvus user/password issue.

invalid auth token is coming from etcd server. did you enable etcd auth? if so on the milvus side you need to config
etcd
auth:
enabled: false # Whether to enable authentication
userName: # username for etcd authentication
password: # password for etcd authentication

[tmnhy]

Configuration that works fine on docker images v. 2.4.x:

embedEtcd.yaml:

listen-client-urls: http://0.0.0.0:2379
advertise-client-urls: http://0.0.0.0:2379
quota-backend-bytes: 4294967296
auto-compaction-mode: revision
auto-compaction-retention: '1000'

user.yaml:

     common:
       security:
         authorizationEnabled: true
         defaultRootPassword: ...
     proxy:
       maxUserNum: 100
       maxRoleNum: 100

milvus  run standalone  1>/dev/null \
--health-cmd="curl -f http://localhost:9091/healthz" \
--health-interval=30s --health-start-period=90s \
--health-timeout=20s --health-retries=3

Are there any changes in 2.5.x that will prevent this from working? Where can I read about it?

[tmnhy]

Also, if I specify the wrong ADMIN_TOKEN, I get an error.

<_InactiveRpcError: StatusCode.UNAUTHENTICATED, auth check failure, please check username and password are correct>

Doesn't this mean that the link with etcd is working correctly?

[xiaofan-luan]

auth check failure. this is the milvus error I guess.

[yanliang567]

/assign @LoveEachDay
/unassign

[jaime0815]

This Milvus error occurs because common.security.authorizationEnabled is enabled. Ensure the correct username and password are set in the connection to resolve this issue.

Also, if I specify the wrong ADMIN_TOKEN, I get an error.

<_InactiveRpcError: StatusCode.UNAUTHENTICATED, auth check failure, please check username and password are correct>

Doesn't this mean that the link with etcd is working correctly?

[tmnhy]

@jaime0815 Yeah, I get it.

This error I gave as an example as the communication between milvus and etcd is working fine.