Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: MacOS binary crashes during e2e tests #39613

Open
1 task done
ColeDearmon-Moore-at opened this issue Jan 30, 2025 · 4 comments
Open
1 task done

[Bug]: MacOS binary crashes during e2e tests #39613

ColeDearmon-Moore-at opened this issue Jan 30, 2025 · 4 comments
Assignees
@ColeDearmon-Moore-at
Labels
kind/bug Issues or changes related a bug triage/needs-information Indicates an issue needs more information in order to work on it.

Comments

@ColeDearmon-Moore-at
Copy link

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: 2.5.4
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): N/A
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus 2.6.0rc44
- OS(Ubuntu or CentOS): MacOs
- CPU/Memory: 16/64 GB
- GPU: no
- Others:

Current Behavior

I'm trying to setup local development using a binary built from source, but when I run e2e tests it crashes giving the following:

[2025/01/28 14:36:20.049 -08:00] [INFO] [rootcoord/broker.go:161] ["watching channels"] [traceID=52e2976e2e7a710261a59f18916a4800] [ts=455633477314281473] [collection=455633452276785476] [vChannels="[by-dev-rootcoord-dml_15_455633452276785476v0]"]
[2025/01/28 14:36:20.049 -08:00] [WARN] [rootcoord/root_coord.go:1267] ["failed to describe collection"] [traceID=543a3de9b329d8761c78db2f661e4a40] [collectionName=collection_count_d04QHPNr] [dbName=default] [id=0] [ts=18446744073709551615] [allowUnavailable=false] [error="collection not found[database=default][collection=collection_count_d04QHPNr]"]
[2025/01/28 14:36:20.049 -08:00] [INFO] [datacoord/services.go:1250] ["receive watch channels request"] [traceID=52e2976e2e7a710261a59f18916a4800] [collectionID=455633452276785476] [channels="[by-dev-rootcoord-dml_15_455633452276785476v0]"] [dbProperties=null]
[2025/01/28 14:36:20.049 -08:00] [INFO] [datacoord/channel_manager.go:242] ["Add channel"] [traceID=52e2976e2e7a710261a59f18916a4800] [channel=by-dev-rootcoord-dml_15_455633452276785476v0]
SIGBUS: bus error
PC=0x19556b14c m=31 sigcode=1 addr=0x0

goroutine 0 gp=0x140055cda40 m=31 mp=0x14001ef6008 [idle]:
runtime: g 0 gp=0x140055cda40: unknown pc 0x19556b14c
stack: frame={sp:0x17d9f1a20, fp:0x0} stack=[0x17d970000,0x17d9f3000)
0x000000017d9f1920:  0x0000600001770108  0x054816507d5e008f
0x000000017d9f1930:  0x000000017d9f1a10  0x0000000000000002
0x000000017d9f1940:  0x0000000000000001  0x0000000000000008
0x000000017d9f1950:  0x0000000110535a30  0x00006000010cf1e0
0x000000017d9f1960:  0x000000017d9f1af8  0x000000017d9f1b00
0x000000017d9f1970:  0x000000017d9f1aa0  0x000000010d9d78a4
0x000000017d9f1980:  0x000000017d9f1b00  0x0000000000000000
0x000000017d9f1990:  0x0000000000000000  0x0000000000000000
0x000000017d9f19a0:  0x0000000000000000  0x000000017d9f1af0
0x000000017d9f19b0:  0x000000017d9f1b60  0x054816507d5e008f
0x000000017d9f19c0:  0x0000000147821000  0x000000000000000a
0x000000017d9f19d0:  0x000000017d9f19e0  0x9a03000188849760
0x000000017d9f19e0:  0x000000017d9f1a20  0x662980018884d48c
0x000000017d9f19f0:  0x000000017d9f1a00  0xef12800188849714
0x000000017d9f1a00:  0x000000017d9f1a20  0x4c4400018884d50c
0x000000017d9f1a10:  0x000000017d9f1a20  0x5a07800188849760
0x000000017d9f1a20: <0x000000017d9f1d20  0xa95200010d8bba4c
0x000000017d9f1a30:  0x00000000ffffffff  0x000000017d9f1a88
0x000000017d9f1a40:  0x000000017d9f1a70  0xca5600019556c000
0x000000017d9f1a50:  0x0000000000000000  0x000000017d9f2950
0x000000017d9f1a60:  0x000000017d9f1fd0  0x000000017d9f26d8
0x000000017d9f1a70:  0x000000017d9f2660  0x403a80010335c9fc
0x000000017d9f1a80:  0x000060000087c610  0x000000017d9f1b01
0x000000017d9f1a90:  0x0000000000000000  0x000000013a0b9e08
0x000000017d9f1aa0:  0x000000010342c5e8  0x000060000066cbc0
0x000000017d9f1ab0:  0x000060000066cb48  0x0000000000000000
0x000000017d9f1ac0:  0x00000000000000a0  0x0000000000000006
0x000000017d9f1ad0:  0x0000000033b5a260  0x0000000000000000
0x000000017d9f1ae0:  0x0000000000000002  0x0000000000000000
0x000000017d9f1af0:  0x0000000000000002  0x0000000000000000
0x000000017d9f1b00:  0x0000000000000000  0x0000320000000000
0x000000017d9f1b10:  0x0000320000000000  0x0000000195567bc0
runtime: g 0 gp=0x140055cda40: unknown pc 0x19556b14c
stack: frame={sp:0x17d9f1a20, fp:0x0} stack=[0x17d970000,0x17d9f3000)
0x000000017d9f1920:  0x0000600001770108  0x054816507d5e008f
0x000000017d9f1930:  0x000000017d9f1a10  0x0000000000000002
0x000000017d9f1940:  0x0000000000000001  0x0000000000000008
0x000000017d9f1950:  0x0000000110535a30  0x00006000010cf1e0
0x000000017d9f1960:  0x000000017d9f1af8  0x000000017d9f1b00
0x000000017d9f1970:  0x000000017d9f1aa0  0x000000010d9d78a4
0x000000017d9f1980:  0x000000017d9f1b00  0x0000000000000000
0x000000017d9f1990:  0x0000000000000000  0x0000000000000000
0x000000017d9f19a0:  0x0000000000000000  0x000000017d9f1af0
0x000000017d9f19b0:  0x000000017d9f1b60  0x054816507d5e008f
0x000000017d9f19c0:  0x0000000147821000  0x000000000000000a
0x000000017d9f19d0:  0x000000017d9f19e0  0x9a03000188849760
0x000000017d9f19e0:  0x000000017d9f1a20  0x662980018884d48c
0x000000017d9f19f0:  0x000000017d9f1a00  0xef12800188849714
0x000000017d9f1a00:  0x000000017d9f1a20  0x4c4400018884d50c
0x000000017d9f1a10:  0x000000017d9f1a20  0x5a07800188849760
0x000000017d9f1a20: <0x000000017d9f1d20  0xa95200010d8bba4c
0x000000017d9f1a30:  0x00000000ffffffff  0x000000017d9f1a88
0x000000017d9f1a40:  0x000000017d9f1a70  0xca5600019556c000
0x000000017d9f1a50:  0x0000000000000000  0x000000017d9f2950
0x000000017d9f1a60:  0x000000017d9f1fd0  0x000000017d9f26d8
0x000000017d9f1a70:  0x000000017d9f2660  0x403a80010335c9fc
0x000000017d9f1a80:  0x000060000087c610  0x000000017d9f1b01
0x000000017d9f1a90:  0x0000000000000000  0x000000013a0b9e08
0x000000017d9f1aa0:  0x000000010342c5e8  0x000060000066cbc0
0x000000017d9f1ab0:  0x000060000066cb48  0x0000000000000000
0x000000017d9f1ac0:  0x00000000000000a0  0x0000000000000006
0x000000017d9f1ad0:  0x0000000033b5a260  0x0000000000000000
0x000000017d9f1ae0:  0x0000000000000002  0x0000000000000000
0x000000017d9f1af0:  0x0000000000000002  0x0000000000000000
0x000000017d9f1b00:  0x0000000000000000  0x0000320000000000
0x000000017d9f1b10:  0x0000320000000000  0x0000000195567bc0

goroutine 25539 gp=0x140055cdc00 m=31 mp=0x14001ef6008 [running, locked to thread]:
runtime.systemstack_switch()
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:200 +0x8 fp=0x14005867b80 sp=0x14005867b70 pc=0x100675d98
runtime.callers(0x10472a5f8?, {0x14005867c28?, 0x10?, 0x10?})
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/traceback.go:1101 +0x78 fp=0x14005867bf0 sp=0x14005867b80 pc=0x100669698
runtime.saveblockevent(0x47c50c, 0xa, 0x140082ad6e0?, 0x3)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/mprof.go:512 +0x88 fp=0x14005867d30 sp=0x14005867bf0 pc=0x10062e898
sync.event(0x106ca7fe0?, 0x140055cdc00?)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/mprof.go:794 +0xbc fp=0x14005867d60 sp=0x14005867d30 pc=0x10067025c
runtime.semrelease1(0x0?, 0x0, 0x1)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:223 +0x170 fp=0x14005867db0 sp=0x14005867d60 pc=0x100650790
sync.runtime_Semrelease(0x14005867e08?, 0x98?, 0x14005867e28?)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/sema.go:72 +0x1c fp=0x14005867de0 sp=0x14005867db0 pc=0x100672fcc
sync.(*Mutex).unlockSlow(0x14005867e01?, 0x55cdc00?)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:246 +0xd0 fp=0x14005867e10 sp=0x14005867de0 pc=0x10069f100
sync.(*Mutex).Unlock(...)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/sync/mutex.go:223
github.com/milvus-io/milvus/internal/util/cgo.unlockMutex(...)
	/Users/linjie.ding/Desktop/milvus/internal/util/cgo/futures.go:40
_cgoexp_dc5aa59dc932_unlockMutex(0x14005867e88?)
	_cgo_gotypes.go:159 +0x60 fp=0x14005867e30 sp=0x14005867e10 pc=0x102bcdc80
runtime.cgocallbackg1(0x102bcdc20, 0x17d9f2a78, 0x13a0b9e00)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:403 +0x224 fp=0x14005867f00 sp=0x14005867e30 pc=0x1005ff0d4
runtime.cgocallbackg(0x102bcdc20, 0x17d9f2a78, 0x13a0b9e00)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/cgocall.go:322 +0x10c fp=0x14005867f80 sp=0x14005867f00 pc=0x1005fedfc
runtime.cgocallbackg(0x102bcdc20, 0x17d9f2a78, 0x13a0b9e00)
	<autogenerated>:1 +0x1c fp=0x14005867fb0 sp=0x14005867f80 pc=0x10067f10c
runtime.cgocallback(0x14001743ea0, 0x5, 0x14001743fdb)
	/Users/linjie.ding/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_arm64.s:1130 +0xb0 fp=0x14005867fe0 sp=0x14005867fb0 pc=0x100678200

Expected Behavior

No response

Steps To Reproduce

Using this binary: https://drive.google.com/file/d/1HeSdW3Uihxy4FbdlctC0VoYu_vsghnBQ/view?usp=sharing

Milvus Log

milvus_logs.txt

Anything else?

No response
@ColeDearmon-Moore-at ColeDearmon-Moore-at added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 30, 2025
@xiaofan-luan
Copy link
Collaborator

@ColeDearmon-Moore-at
what is the process to create this image?
does this compiled in macos system

@ColeDearmon-Moore-at
Copy link
Author

Yeah, this was built from source on mac.

@xiaofan-luan
Copy link
Collaborator

From the stack
runtime/mprof.go

it seems to be a prof path on ARM. disable http.enablePprof see if it works.

@yanliang567
Copy link
Contributor

/assign @ColeDearmon-Moore-at
/unassign

@yanliang567 yanliang567 added triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ColeDearmon-Moore-at ColeDearmon-Moore-at

Labels
kind/bug Issues or changes related a bug triage/needs-information Indicates an issue needs more information in order to work on it.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yanliang567 @xiaofan-luan @ColeDearmon-Moore-at