Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for v2.9.2, v2.10.1, and v2.10.3 and paging support for large metadata #38

Merged
merged 21 commits into from
Feb 14, 2025

Conversation

crupakheti
Copy link
Contributor

@crupakheti crupakheti commented Feb 14, 2025

What type of PR is this? (check all applicable)

  • Refactor
  • Feature
  • Bug Fix
  • Optimization
  • Documentation Update
  • Dependency Update
  • Release

Description

This PR makes following contribution:

  • Support for MWAA v2.9.2, due to gracious contribution of holly-evans 🙏 🚀
  • Support for MWAA v2.10.1 and v2.10.3, due to gracious contributions of pyrr and holly-evans 🙏 🚀
  • Performance enhancement for large metadata export to S3 using server-side cursor
  • Performance enhancement for large metadata import using batched copy into meta-database, thanks to the gracious contribution of mathiasflorin 🙏 🚀

Related Tickets & Documents

Linting

Have you done linting by issuing ./build.sh lint command?

  • Yes
  • No, I need help with linting

Testing

Have you run testing by issuing ./build.sh unit command?

  • Yes
  • No, I need help with writing tests

Documentation

Have you updated the README appropriate for this PR?

  • Yes
  • No, README does not need any changes for this PR

Changelog

Have you updated the CHANGELOG with the changes you are making in this PR?
Please use Added, Removed, and/or Changed subsections under the Unreleased section.

  • Yes
  • No, I need help

Terms of Contribution

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

mathiasflorin and others added 20 commits November 20, 2024 16:42
dr_factory.py: Filters backup_metadata to ensure the scheduler does not resume backup_metadata with the status running during batch imports of metadata. If the scheduler resumes, it will overwrite the metadata while the import is running.
test_dr_factory_2_4.py: Add expected_export_filters to the tests.

test_base_table.py:

Adjust tests to assert that copy_expert is called with StringIO("".join(batch)).

Add test_restore_batch_size to ensure that commit is called twice when the file contains more rows than the batch size.
…store_more_than_batch_size and test_restore_cursor_exception
… itertools.islice for yielding lines instead of looping over the lines of the file. Thank you for the recommendation!
…sor/paging to reduce the memory pressure. Fixing broken test cases.
@crupakheti crupakheti added the enhancement New feature or request label Feb 14, 2025
@crupakheti crupakheti self-assigned this Feb 14, 2025
Copy link

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  .
  config.py
  assets/dags/mwaa_dr/framework/factory
  base_dr_factory.py
  default_dag_factory.py
  assets/dags/mwaa_dr/framework/model
  active_dag_table.py
  base_table.py
  connection_table.py
  variable_table.py
  assets/dags/mwaa_dr/v_2_10
  dr_factory.py
  assets/dags/mwaa_dr/v_2_9
  dr_factory.py
  lib/dr_constructs
  airflow_cli.py
  lib/functions
  airflow_cli_client.py
  airflow_cli_function.py
  lib/stacks
  mwaa_base_stack.py
  mwaa_primary_stack.py
  mwaa_secondary_stack.py
Project Total  

This report was generated by python-coverage-comment-action

@crupakheti crupakheti merged commit 7d2e9b4 into main Feb 14, 2025
5 checks passed
@crupakheti crupakheti deleted the feature-2-9-2-through-2-10-3 branch February 14, 2025 00:50
@crupakheti crupakheti mentioned this pull request Feb 14, 2025
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEATURE] Support for MWAA v2.9.2
3 participants