Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Benchmarks #264

Merged
merged 7 commits into from
Jan 4, 2024
Merged

Update Benchmarks #264

merged 7 commits into from
Jan 4, 2024

Conversation

smohan-dw
Copy link
Member

No description provided.

@smohan-dw
Copy link
Member Author

Need to run this again on a machine with an NvEME disk and a CPU with better single-thread performance.

[-] Failed the machine benchmark:\n2024-01-02 18:40:21 Running machine benchmarks...
2024-01-02 18:40:47
+----------+----------------+-------------+-------------+-------------------+
| Category | Function | Score | Minimum | Result |
+===========================================================================+
| CPU | BLAKE2-256 | 863.60 MiBs | 783.27 MiBs | ✅ Pass (110.3 %) |
|----------+----------------+-------------+-------------+-------------------|
| CPU | SR25519-Verify | 402.96 KiBs | 560.67 KiBs | ❌ Fail ( 71.9 %) |
|----------+----------------+-------------+-------------+-------------------|
| Memory | Copy | 4.52 GiBs | 11.49 GiBs | ❌ Fail ( 39.4 %) |
|----------+----------------+-------------+-------------+-------------------|
| Disk | Seq Write | 240.02 MiBs | 950.00 MiBs | ❌ Fail ( 25.3 %) |
|----------+----------------+-------------+-------------+-------------------|
| Disk | Rnd Write | 119.94 MiBs | 420.00 MiBs | ❌ Fail ( 28.6 %) |
+----------+----------------+-------------+-------------+-------------------+
From 5 benchmarks in total, 1 passed and 4 failed (10% fault tolerance).

amarts
amarts previously approved these changes Jan 3, 2024
Copy link
Member

@amarts amarts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Few question. The benchmark runs, should we do that in CI machines? (and thus make this CI fail depending on the type of machine we get in github?) or is there an option to run it specifically on some setup with docker, so we can ask output of this command to be sent to us before starting a validator node ?

// Measured: `661`
// Estimated: `19007`
// Minimum execution time: 55_169_000 picoseconds.
Weight::from_parts(57_651_421, 19007)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one place where we are passing variable length argument. Should the weight change based on that?

@@ -766,7 +766,7 @@ pub mod pallet {
/// the number of entries
/// removed.
#[pallet::call_index(4)]
#[pallet::weight(<T as pallet::Config>::WeightInfo::remove( ))]
#[pallet::weight(<T as pallet::Config>::WeightInfo::remove(T::MaxRemoveEntries::get() as u32))]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would this require a version change?

/// Average: 223_580
/// Median: 221_485
/// Std-Dev: 5572.62
/// Min, Max: 280_638, 577_472
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Compared to earlier a lot of diff between min and max. Is this alright?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Average has not moved up much. Couple of reasons, increased blocktime and WASM

@smohan-dw smohan-dw marked this pull request as ready for review January 3, 2024 13:36
@amarts
Copy link
Member

amarts commented Jan 4, 2024

For those who are referring, the hardware tests are not part of this PR, but will be added back as a separate PR, so we can discuss more on it with focus.

@smohan-dw smohan-dw merged commit c2c8f75 into dhiway:develop Jan 4, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants