Improve src net rucio script #612
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #612 +/- ##
=======================================
Coverage 68.62% 68.63%
=======================================
Files 53 53
Lines 5696 5713 +17
=======================================
+ Hits 3909 3921 +12
- Misses 1787 1792 +5 ☔ View full report in Codecov by Sentry. |
sfiruch
requested changes
Aug 22, 2024
|
@sfiruch I've tested the script and it worked. However, there was an odd |
Lukas113
force-pushed
the
improve-SRCNet-rucio-script
branch
from
41013ba to
a17e4ab
Compare
sfiruch
approved these changes
Sep 16, 2024
mpluess
pushed a commit
that referenced
this pull request
Sep 24, 2024
* init commit to extract casa ms metadata using casacore 🌸 * bugfix: dtypes of Observation Table ♊ * added polarization table to casa module 🌞 * added antenna table to casa MS meta 🚨 * Improve src net rucio script (#612) * init commit of improved srcnet rucio script 🍒 * bugfix: calc img-cellsize 🚖 * minor bugfix and improvements 🎾 * renamed longest-baseline function to max-baseline 🉐 * calculated image parameters from simulation observation in demo script ♨️ * implemented get-corners-in-world in Image class 😘 * bugfix: calc ObsCore region coverage on sphere 🍑 * bugfix: added obs-publisher-did to restored obscore 🐟 * replaced SRCNet-rucio-meta with demo-script 👜 * addressed PR requests 🌿 * addressed PR requests ✋ * removed inc-freq-hz check in Observation init 💤 * specified mypy ignore comment in download◀️ * added main table to casa module 💖 * added field table to casa module 🍩 * implemented spectral-window in casa metadata 🔍 * removed main table from casa meta reading to avoid expensive loading of each measurement ⛅ * intermediate commit change from-visibility to MS 🎽 * added get-col to casa ABC class ♒ * implemented nrows to casa ABC 👮 * deatched get-baselines-dists from self ⏳ * switched ObsCoreMeta.from-visibility to .ms from .vis 🈂️ * Fix error with wrong FITS header card (#618) * Fix error with wrong FITS header card * Silence black formatter * implemented no polarization option into ObsCoreMata.from-visibility 🚭 * addressed merged changes 💣 * readded oskar-vis support to ObsCoreMeta.from-visibility 💜 * minor docstring fixes 🚿 * added MS-version extraction utility function to casa.py 🉐 * renamed casa ms-version properly 🍉 * addressed PR requests 🙏 * addressed MR changes 🔁 --------- Co-authored-by: Andreas Wassmer <[email protected]>
mpluess
added a commit
that referenced
this pull request
Oct 9, 2024
* OSKAR: only write visibilities in one format, not both. RASCIL: changed default visibility format to MS. Made visibility format configurable. WIP, tests TODO. 🚂 * Copied export_visibility_to_ms function from ska_sdp_datamodels to our code base to be able to use it without upgrading ska_sdp_datamodels + RASCIL + Python. 🐛 * Source, license info added for ska_sdp_datamodels code 😟 * Renamed VIS visibility format to OSKAR_VIS to make it more clear. Removed HDF5 export for RASCIL to avoid having a third format. 🐣 * Visibility: removed unnecessary read and write methods. Refactored path validation. 🎫 * Allow user to pass visibility format themselves (mostly to anticipate future formats like MSv3, DASK-MS where automatic parsing might not be possible) 🐯 * Remove faulty test and (otherwise) unused methods 🏧 * Refactored combine_vis. Shortened Visibility class attribute names. 🐸 * Explicitly type Visibility.format 💲 * Refactored ms_file_path, vis_path in Visibility, Interferometer classes* Always return Visibility objects from simulations, not RASCILVisibility 🌕 * Enable obscore for casa ms (#619) * init commit to extract casa ms metadata using casacore 🌸 * bugfix: dtypes of Observation Table ♊ * added polarization table to casa module 🌞 * added antenna table to casa MS meta 🚨 * Improve src net rucio script (#612) * init commit of improved srcnet rucio script 🍒 * bugfix: calc img-cellsize 🚖 * minor bugfix and improvements 🎾 * renamed longest-baseline function to max-baseline 🉐 * calculated image parameters from simulation observation in demo script ♨️ * implemented get-corners-in-world in Image class 😘 * bugfix: calc ObsCore region coverage on sphere 🍑 * bugfix: added obs-publisher-did to restored obscore 🐟 * replaced SRCNet-rucio-meta with demo-script 👜 * addressed PR requests 🌿 * addressed PR requests ✋ * removed inc-freq-hz check in Observation init 💤 * specified mypy ignore comment in download
◀️ * added main table to casa module 💖 * added field table to casa module 🍩 * implemented spectral-window in casa metadata 🔍 * removed main table from casa meta reading to avoid expensive loading of each measurement ⛅ * intermediate commit change from-visibility to MS 🎽 * added get-col to casa ABC class ♒ * implemented nrows to casa ABC 👮 * deatched get-baselines-dists from self ⏳ * switched ObsCoreMeta.from-visibility to .ms from .vis 🈂️ * Fix error with wrong FITS header card (#618) * Fix error with wrong FITS header card * Silence black formatter * implemented no polarization option into ObsCoreMata.from-visibility 🚭 * addressed merged changes 💣 * readded oskar-vis support to ObsCoreMeta.from-visibility 💜 * minor docstring fixes 🚿 * added MS-version extraction utility function to casa.py 🉐 * renamed casa ms-version properly 🍉 * addressed PR requests 🙏 * addressed MR changes 🔁 --------- Co-authored-by: Andreas Wassmer <[email protected]> * Started imager interface refactoring 📋 * Refactored dirty imaging and image cleaning interfaces and implementations to only accept Karabo Visibility objects 🔡 * Cleaned up run_simulation overloads. Removed format parameter in Visibility constructor. Avoid typing errors. 😩 * Made visibility arguments positional only in imaging methods 😹 * Fixed line emission visibility file path ✋ * Source detection: re-added comparison of source detection on dirty and cleaned image. Improved titles, description. 🔗 * Removed imaging module page from docs, was showing an empty page ⛎ * run_sample_simulation: make simulator backend, visibility format configurable, return all relevant objects. Visibility: verify path in constructor. Added test for simulation-format-imager combos. 🍥 * Improved WSClean tests 🚬 * Docstrings 🐊 * codecov: ignore temporary copy of ska_sdp_datamodels MS functionality 🏇 * adapted tests for obscore casa-ms and casa module itself 🎶 * Fixed mypy errors* Fixed run_sample_simulation call 💣 * Fixed imaging notebook 📋 * Addressed review comments part 1 📝 * Addressed review comments part 2. Changed the clean_disk fixture during tests to session scope. 🍄 * Addressed review comments part 3 🎣 * default_sample_simulation_visibility needs same scope as disk cleanup function 📄 * Added casacore direct dependency ⬛ --------- Co-authored-by: Lukas Gehrig <[email protected]> Co-authored-by: Andreas Wassmer <[email protected]> Co-authored-by: lukas.gehrig <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #611
According to the SRCNet efforts, a demo to create ObsCore metadata and ingest it into the ingestion daemon is needed. Therefore I adapted the
SRCNet_rucio_meta.pyexample script to be a full simulation. This will be helpful for future dataset creation.In addition, I also discovered a bug by accident, which I described in #611 and is solved in this PR.