Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(i): Test benchmark current local acp operations #3396

Merged
merged 3 commits into from
Jan 22, 2025

Conversation

shahzadlone
Copy link
Member

@shahzadlone shahzadlone commented Jan 20, 2025

Relevant issue(s)

Resolves #3389

Description

  • Bench mark acp register call
  • Bench mark acp check call
  • Bench mark acp is object registered? call

Benchmarking Reports:

TLDR Comparisons of Old vs New:

Register:

BenchmarkACPRegisterOld/scale=256,inMem=true-22       14     81.2 ms/op     62.8 MB/op       1M allocs/op
BenchmarkACPRegisterOld/scale=512,inMem=true-22        4      283 ms/op      229 MB/op       4M allocs/op
BenchmarkACPRegisterOld/scale=1024,inMem=true-22       1      1.07 s/op      871 MB/op      15M allocs/op
BenchmarkACPRegisterOld/scale=2048,inMem=true-22       1      4.00 s/op     3.41 GB/op      60M allocs/op
BenchmarkACPRegisterOld/scale=4096,inMem=true-22       1      16.1 s/op     13.5 GB/op     238M allocs/op
BenchmarkACPRegisterOld/scale=8192,inMem=true-22       1      68.9 s/op     53.9 GB/op     946M allocs/op

BenchmarkACPRegisterNew/scale=256,inMem=true-22       57     21.8 ms/op     11.6 MB/op     213k allocs/op
BenchmarkACPRegisterNew/scale=512,inMem=true-22       25     44.5 ms/op     23.1 MB/op     426k allocs/op
BenchmarkACPRegisterNew/scale=1024,inMem=true-22      13     88.7 ms/op     46.3 MB/op     852k allocs/op
BenchmarkACPRegisterNew/scale=2048,inMem=true-22       6      178 ms/op     92.6 MB/op       2M allocs/op
BenchmarkACPRegisterNew/scale=4096,inMem=true-22       3      364 ms/op      185 MB/op       3M allocs/op
BenchmarkACPRegisterNew/scale=8192,inMem=true-22       2      744 ms/op      371 MB/op       7M allocs/op

CheckDocAccess:

BenchmarkACPCheckDocAccessOld/scale=256,inMem=true-22       10     126 µs/op     34.9 kB/op     772 allocs/op
BenchmarkACPCheckDocAccessOld/scale=512,inMem=true-22       10     160 µs/op     34.9 kB/op     772 allocs/op
BenchmarkACPCheckDocAccessOld/scale=1024,inMem=true-22      10     138 µs/op     34.9 kB/op     772 allocs/op
BenchmarkACPCheckDocAccessOld/scale=2048,inMem=true-22      10     177 µs/op     34.9 kB/op     772 allocs/op
BenchmarkACPCheckDocAccessOld/scale=4096,inMem=true-22      10     171 µs/op     35.0 kB/op     773 allocs/op
BenchmarkACPCheckDocAccessOld/scale=8192,inMem=true-22      10     216 µs/op     36.2 kB/op     774 allocs/op

BenchmarkACPCheckDocAccessNew/scale=256,inMem=true-22       10     101 µs/op     34.7 kB/op     769 allocs/op
BenchmarkACPCheckDocAccessNew/scale=512,inMem=true-22       10     102 µs/op     34.8 kB/op     770 allocs/op
BenchmarkACPCheckDocAccessNew/scale=1024,inMem=true-22      10     101 µs/op     34.9 kB/op     771 allocs/op
BenchmarkACPCheckDocAccessNew/scale=2048,inMem=true-22      10     101 µs/op     34.9 kB/op     771 allocs/op
BenchmarkACPCheckDocAccessNew/scale=4096,inMem=true-22      10     195 µs/op     34.9 kB/op     772 allocs/op
BenchmarkACPCheckDocAccessNew/scale=8192,inMem=true-22      10     102 µs/op     34.9 kB/op     772 allocs/op

IsDocRegistered:

BenchmarkACPIsDocRegisteredOld/scale=256,inMem=true-22       10      249 µs/op      221 kB/op      4k allocs/op
BenchmarkACPIsDocRegisteredOld/scale=512,inMem=true-22       10      484 µs/op      422 kB/op      8k allocs/op
BenchmarkACPIsDocRegisteredOld/scale=1024,inMem=true-22      10      957 µs/op      836 kB/op     15k allocs/op
BenchmarkACPIsDocRegisteredOld/scale=2048,inMem=true-22      10     2.42 ms/op     1.64 MB/op     29k allocs/op
BenchmarkACPIsDocRegisteredOld/scale=4096,inMem=true-22      10     3.98 ms/op     3.32 MB/op     58k allocs/op
BenchmarkACPIsDocRegisteredOld/scale=8192,inMem=true-22      10     9.00 ms/op     6.53 MB/op    115k allocs/op

BenchmarkACPIsDocRegisteredNew/scale=256,inMem=true-22       10     54.6 µs/op     20.9 kB/op     395 allocs/op
BenchmarkACPIsDocRegisteredNew/scale=512,inMem=true-22       10     49.1 µs/op     20.9 kB/op     395 allocs/op
BenchmarkACPIsDocRegisteredNew/scale=1024,inMem=true-22      10     44.3 µs/op     20.9 kB/op     395 allocs/op
BenchmarkACPIsDocRegisteredNew/scale=2048,inMem=true-22      10     50.0 µs/op     20.9 kB/op     395 allocs/op
BenchmarkACPIsDocRegisteredNew/scale=4096,inMem=true-22      10     47.0 µs/op     20.9 kB/op     395 allocs/op
BenchmarkACPIsDocRegisteredNew/scale=8192,inMem=true-22      10     50.1 µs/op     20.9 kB/op     395 allocs/op

@shahzadlone shahzadlone added perf Performance issue or suggestion area/testing Related to any test or testing suite labels Jan 20, 2025
@shahzadlone shahzadlone added this to the DefraDB v0.16 milestone Jan 20, 2025
@shahzadlone shahzadlone self-assigned this Jan 20, 2025
@shahzadlone shahzadlone requested a review from Lodek January 20, 2025 17:27
@shahzadlone shahzadlone force-pushed the lone/bench/acp-core branch 5 times, most recently from 3a89bf5 to ecc4a51 Compare January 20, 2025 18:51
@shahzadlone shahzadlone marked this pull request as ready for review January 20, 2025 19:07
@shahzadlone shahzadlone requested a review from a team January 20, 2025 19:07
Copy link

codecov bot commented Jan 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 78.34%. Comparing base (727800a) to head (65584d6).
Report is 1 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3396      +/-   ##
===========================================
+ Coverage    78.14%   78.34%   +0.21%     
===========================================
  Files          392      392              
  Lines        36045    36045              
===========================================
+ Hits         28164    28239      +75     
+ Misses        6201     6149      -52     
+ Partials      1680     1657      -23     
Flag Coverage Δ
all-tests 78.34% <ø> (+0.21%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 16 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 727800a...65584d6. Read the comment docs.

@shahzadlone shahzadlone force-pushed the lone/bench/acp-core branch 4 times, most recently from 0dae79d to ba38cb4 Compare January 20, 2025 21:58
Copy link
Collaborator

@fredcarle fredcarle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one todo to remove commented out code.

One thing that would be nice is maybe instead of having the commit as part of the title, I would prefer having the date of the commit (acp_bench_report_20250114.md) so that the files can be ordered sequentially. The commit is mentioned at the top of the documentation so it will always be easy to refer to it if needed.

DID: "did:key:z7r8os2G88XXBNBTLj3kFR5rzUJ4VAesbX7PgsA68ak9B5RYcXF5EZEmjRzzinZndPSSwujXb4XKHG6vmKEFG6ZfsfcQn",
}

//var identity2 = identity.Identity{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: Please remove commented code

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to keep it for now. I was benching a few other acp ops that I need it for (relationship stuff).

Copy link
Member

@Lodek Lodek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, nice job Shahzad

@shahzadlone
Copy link
Member Author

I would prefer having the date of the commit (acp_bench_report_20250114.md) so that the files can be ordered sequentially

I like this suggestion, will do!

@shahzadlone shahzadlone merged commit 1da6462 into sourcenetwork:develop Jan 22, 2025
42 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/testing Related to any test or testing suite perf Performance issue or suggestion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Investigate / Benchmark ACP Core Register and Check Calls
3 participants