Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v5.0.0 #181

Merged
merged 38 commits into from
Dec 30, 2024
Merged

v5.0.0 #181

merged 38 commits into from
Dec 30, 2024

Conversation

Frix-x
Copy link
Owner

@Frix-x Frix-x commented Dec 22, 2024

Shake&Tune v5.0.0 is a big release with many changes to the internal code, adding a dedicated .stdata format, a new sweep test (compatibility with the latest Klipper release). It also includes various improvements to mitigate "timer too close" errors on lower-end hardware, and a refactoring of the graph generation code for future extensibility, etc.

Changes

  • Measurements are now stored in a new custom .stdata format, giving more control over what is saved to disk and how it is saved. This is easily extensible to store more things in the future. This new approach to data handling also reduces the chance of "timer too close" errors, reducing problems on less powerful hosts (e.g. CB1). But it's still not perfect, so if you still encounter problems, try stopping Crowsnest, Neopixels, etc. before running a test...

  • Fixes the bug introduced with the latest Klipper version ('ResonanceTester' object has no attribute 'test' (latest Klipper break Shake&Tune) #172). Now Shake&Tune is compatible with the new sweep test, allowing pulse-only (original test) with or without additional motion sweeps to get better graphs, especially on less rigid machines. Please refer to the relevant part of the documentation to choose which test to use and when. In the same vein, the graph headers have also been updated to show what type of test was used and the associated parameters.

  • Introduced a proper command line interface (CLI) for generating graphs from stored data, with support for the new compressed .stdata file format, but also compatible with traditional .csv files. This is super useful for me to debug new features offline, but you can also use it to manually (re)generate some graphs from the CSV you saved.

  • The plotting code has been completely rewritten and refactored. The plotting functions are now separated from the data calculations, making it easier to add new tests or switch to a different plotting backend in the future (and yes, I do have plans for that... Stay tuned!).

  • I've also added some draft code for a future Acceleration vs. Vibration vs. Smoothing trade-off plot. But the current plotting backend doesn't allow all the things I want to make it easy to read, so the code is there but disabled in this release until I find a proper way to present the data.

  • Added compatibility with the "limited" kinematics of Danger-Klipper / Kalico.

  • An optional MAX_SCALE= parameter has been added to the macros to force the energy scale on the belts and input shaper graphs to certain values. This makes it easier to compare several consecutive runs by viewing the graphs side by side with the same scale.

Miscellaneous fixes

  • Improve the startup logic to fail fast during Klipper initialization if a configuration problem is detected. This did not work properly in the past and led to problems where e.g. the [resonance_tester] section was missing but remained undetected and led to a crash when running a test.

  • Updated CI smoke tests to run against better Klipper mockups using a fake board and kinematics instead of the previously used kinematics: none and also by trying against multiple Python and Klipper versions.

  • Fixed incorrect marker colors in the cross-belt plots (purple/orange points were inverted on this plot).

  • Increased the default timeout to 10 minutes, as the vibrations graph can take a long time to run, and some people were experiencing frequent failures due to this.

  • Since the .stdata file format is added, I now have full control over how and when the files are written to disk. So now Shake&Tune will also wait for the file writing to be finished before starting a new accelerometer measurement. This helps with timer too close errors, but also fixes a rare bug where the vibrations graph would not completely record all the data for one of the motors.

  • Added support for alternative Klippy venv in case you don't have a standard installation

Frix-x and others added 30 commits July 13, 2024 11:04
Colors were previously inverted on the left graph but it was forgotten to also do it on the versus plot...
accel vs vibrs performance plot added
to avoid having multiple CSV files open at once
and accelerometer measurements on-going at the same time
feat(kinematics): add support for danger-klipper limited_ kinematics
feat(ci): expand matrix accross python and klipper versions
with all the logic to load them back and reconstruct the .stdata file
with all the logic to load them back and reconstruct the .stdata file
Adding the chunk size as a shaketune parameter
Support alternate klipper venv paths
Trying to fix Klipper TTC errors
* implemented a Factory design pattern for graph creators

* factorized Klipper commands

* separation of computations vs plotters

* simplify logical expressions using De Morgan identities

* replace mutable default logo position argument with None

* inline computation results in the return statement directly

* using union operator for the graph creators parameters

* use next() to return the formatted vector in axes map function

* using walrus operator to simplify logic in GraphCreatorFactory
* small refactoring of k_shapers imports and variables to be easier to read
* added draft for tradeoff shaper plots (commented for now as it's a WIP for later use)
* fixed compatibility with the new version of Klipper for pulse-only test

* added sweeping compatibility (#179)

* added documentation about sweeping mode
Copy link
Contributor

sourcery-ai bot commented Dec 22, 2024

Reviewer's Guide by Sourcery

This PR refactors the graph_creators module to use a more robust and efficient data handling approach. It introduces a MeasurementsManager class to handle accelerometer data, replacing the previous reliance on temporary CSV files. The plotting logic is extracted into a dedicated Plotter class, and a factory pattern is implemented for creating graph creators. Additionally, support for a command-line interface (CLI) is added.

Sequence diagram for graph creation process

sequenceDiagram
    participant User
    participant ShakeTune
    participant GraphCreatorFactory
    participant GraphCreator
    participant MeasurementsManager

    User->>ShakeTune: Execute command
    ShakeTune->>GraphCreatorFactory: create_graph_creator(type, config)
    GraphCreatorFactory-->>ShakeTune: graph_creator
    ShakeTune->>GraphCreator: configure(params)
    ShakeTune->>MeasurementsManager: collect measurements
    MeasurementsManager-->>ShakeTune: measurements
    ShakeTune->>GraphCreator: create_graph(measurements_manager)
    GraphCreator->>GraphCreator: compute()
    GraphCreator->>GraphCreator: plot()
    GraphCreator-->>ShakeTune: figure
    ShakeTune-->>User: Save graph
Loading

Class diagram showing the new graph creator architecture

classDiagram
    class GraphCreator {
        <<abstract>>
        +configure()
        +create_graph(measurements_manager)
    }

    class GraphCreatorFactory {
        +create_graph_creator(type, config)
    }

    class VibrationsGraphCreator {
        -_kinematics: str
        -_accel: float
        -_motors: List[Motor]
        +configure(kinematics, accel, motor_config_parser)
        +create_graph(measurements_manager)
    }

    class BeltsGraphCreator {
        -_kinematics: str
        -_accel_per_hz: float
        +configure(kinematics, accel_per_hz)
        +create_graph(measurements_manager)
    }

    class StaticGraphCreator {
        -_freq: float
        -_duration: float
        -_accel_per_hz: float
        +configure(freq, duration, accel_per_hz)
        +create_graph(measurements_manager)
    }

    GraphCreator <|-- VibrationsGraphCreator
    GraphCreator <|-- BeltsGraphCreator
    GraphCreator <|-- StaticGraphCreator
    GraphCreatorFactory ..> GraphCreator : creates
Loading

Class diagram showing the new measurement management system

classDiagram
    class MeasurementsManager {
        -measurements: List[Measurement]
        +get_measurements()
        +add_measurement(measurement)
    }

    class Measurement {
        +name: str
        +samples: List
    }

    class ShakeTuneConfig {
        +result_folder: Path
        +keep_n_results: int
        +keep_raw_data: bool
        +chunk_size: int
        +max_freq: float
        +dpi: int
    }

    MeasurementsManager --> "*" Measurement : manages
    MeasurementsManager --> "1" ShakeTuneConfig : uses
Loading

File-Level Changes

Change Details Files
Introduce MeasurementsManager to handle accelerometer data, replacing temporary CSV files.
  • Implement MeasurementsManager to store and manage accelerometer measurements.
  • Refactor data loading and saving to use MeasurementsManager.
  • Remove temporary CSV file handling.
  • Add support for compressed .stdata files.
  • Implement chunking of measurements to avoid memory issues.
  • Update commands to use MeasurementsManager.
shaketune/graph_creators/vibrations_graph_creator.py
shaketune/graph_creators/belts_graph_creator.py
shaketune/graph_creators/shaper_graph_creator.py
shaketune/graph_creators/axes_map_graph_creator.py
shaketune/helpers/resonance_test.py
shaketune/commands/compare_belts_responses.py
shaketune/commands/axes_shaper_calibration.py
shaketune/commands/axes_map_calibration.py
shaketune/commands/create_vibrations_profile.py
shaketune/commands/excitate_axis_at_freq.py
shaketune/shaketune_process.py
shaketune/helpers/accelerometer.py
shaketune/graph_creators/static_graph_creator.py
Extract plotting logic into a dedicated Plotter class.
  • Create Plotter class to handle graph creation and plotting.
  • Move plotting functions from individual graph creators to Plotter.
  • Update graph creators to use Plotter for plotting.
shaketune/graph_creators/vibrations_graph_creator.py
shaketune/graph_creators/belts_graph_creator.py
shaketune/graph_creators/shaper_graph_creator.py
shaketune/graph_creators/axes_map_graph_creator.py
shaketune/graph_creators/static_graph_creator.py
shaketune/graph_creators/plotter.py
Implement factory pattern for creating graph creators.
  • Create GraphCreatorFactory to manage graph creator instantiation.
  • Register graph creators with the factory.
  • Update ShakeTune to use the factory for creating graph creators.
shaketune/shaketune.py
shaketune/graph_creators/graph_creator.py
shaketune/graph_creators/__init__.py
shaketune/graph_creators/graph_creator_factory.py
Add command-line interface (CLI) support.
  • Implement CLI using argparse.
  • Add support for loading data from files in CLI.
  • Enable overriding output target in CLI.
shaketune/cli.py
shaketune/graph_creators/graph_creator.py
Update resonance test logic and add support for sweeping tests.
  • Refactor resonance test logic to handle both old and new Klipper versions.
  • Add support for sweeping tests.
  • Update commands to use the new resonance test logic.
shaketune/helpers/resonance_test.py
shaketune/commands/compare_belts_responses.py
shaketune/commands/axes_shaper_calibration.py
shaketune/commands/create_vibrations_profile.py
shaketune/commands/excitate_axis_at_freq.py
Update configuration handling and add new options.
  • Add new configuration options for data handling and graph generation.
  • Update documentation to reflect the new options.
  • Refactor configuration initialization.
shaketune/shaketune.py
shaketune/shaketune_config.py
README.md
Update documentation and examples.
  • Update documentation to reflect the changes in the code.
  • Add examples for using the CLI.
  • Clarify the usage of sweeping tests.
README.md
docs/is_tuning_generalities.md
Refactor code for improved readability and maintainability.
  • Improve code structure and organization.
  • Add comments and docstrings.
  • Simplify complex functions.
  • Remove unused code.
shaketune/graph_creators/vibrations_graph_creator.py
shaketune/graph_creators/belts_graph_creator.py
shaketune/graph_creators/shaper_graph_creator.py
shaketune/graph_creators/axes_map_graph_creator.py
shaketune/helpers/resonance_test.py
shaketune/commands/compare_belts_responses.py
shaketune/commands/axes_shaper_calibration.py
shaketune/commands/axes_map_calibration.py
shaketune/commands/create_vibrations_profile.py
shaketune/commands/excitate_axis_at_freq.py
shaketune/shaketune_process.py
shaketune/helpers/accelerometer.py
shaketune/graph_creators/static_graph_creator.py
shaketune/shaketune.py
shaketune/shaketune_config.py
README.md
docs/is_tuning_generalities.md
requirements.txt
.github/workflows/test.yml
install.sh
shaketune/graph_creators/graph_creator.py
shaketune/graph_creators/__init__.py
shaketune/graph_creators/plotter.py
shaketune/graph_creators/graph_creator_factory.py
MAINTNENCE.md

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time. You can also use
    this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@Frix-x Frix-x added the enhancement New feature or request label Dec 22, 2024
Copy link
Contributor

@sourcery-ai sourcery-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @Frix-x - I've reviewed your changes and they look great!

Here's what I looked at during the review
  • 🟡 General issues: 3 issues found
  • 🟢 Security: all looks good
  • 🟡 Testing: 3 issues found
  • 🟡 Complexity: 3 issues found
  • 🟢 Documentation: all looks good

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

shaketune/helpers/accelerometer.py Show resolved Hide resolved
shaketune/graph_creators/plotter.py Show resolved Hide resolved
shaketune/helpers/resonance_test.py Show resolved Hide resolved
shaketune/helpers/resonance_test.py Outdated Show resolved Hide resolved
shaketune/helpers/resonance_test.py Outdated Show resolved Hide resolved
shaketune/helpers/accelerometer.py Show resolved Hide resolved
shaketune/helpers/resonance_test.py Show resolved Hide resolved
Frix-x and others added 3 commits December 22, 2024 22:35
Co-authored-by: sourcery-ai[bot] <58596630+sourcery-ai[bot]@users.noreply.github.com>
Copy link

@akilgore8484 akilgore8484 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Everything looks good

Copy link

@jpapiez jpapiez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated readmes look good.

Copy link

@HeavilyModdedEnder3 HeavilyModdedEnder3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. I switched over to this PR, and AXES_SHAPER_CALIBRATION does the sweeping motion

@Frix-x
Copy link
Owner Author

Frix-x commented Dec 29, 2024

Ok, now I think I'm ready to merge v5.0.0 with the couple of additions from tonight (new MAX_SCALE parameter and more infos in the header about the current test mode). Can you all confirm that it's still good for you?

@akilgore8484 @jpapiez @HeavilyModdedEnder3

@roykrikke
Copy link

Will you also include this in the release? #140

@akilgore8484
Copy link

Looks good. Like the additional info in the header. Very helpful. I have the Dev branch running on three different machines with no issues on two of them. On the third machine I've switched back to traditional "no sweep" behavior because the sweep speed of 60mm/s just happens to be where my motors resonate the most. I know mine is an edge case, but would be nice to be able to adjust the sweep speed. It has a sharp resonate peak between 57 and 73 mm/s.

@Frix-x
Copy link
Owner Author

Frix-x commented Dec 30, 2024

Will you also include this in the release? #140

That's a Klippain config issue, but I will switch to it after v5.0.0 of S&T is released and this will likely be part of it

I know mine is an edge case, but would be nice to be able to adjust the sweep speed

This can already be done by tweaking the sweeping_period of the [resonance_tester] config entry. If you decrease the period, I think it will go faster.
In Shake&Tune, I usually try to avoid duplicating config entries and use the ones that are in the stock Klipper if available to avoid confusion. But I may try to find a way to force it using a macro parameter and calculate the new period to display it on the graph, but if not set, default to the original sweeping_period entry. But I need to think about it a bit more to try to find the best and simplest way to do this and it will probably be for a v5.1.0 release.

@Frix-x Frix-x merged commit 74364d7 into main Dec 30, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
9 participants