Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine low level moisture with a cloud mask #3022

Open
wants to merge 18 commits into
base: main
Choose a base branch
from

Conversation

gerritholl
Copy link
Member

@gerritholl gerritholl commented Dec 18, 2024

Combine ESSL colorized low level moisture with a cloud mask.

Example for eurol:

MTG-I1-fci-eurol-colorized_low_level_moisture_with_vis06-20241217120000-20241217121000

Combine ESSL colorized low level moisture with a cloud mask.

Work in progress.
Copy link

codecov bot commented Dec 18, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.11%. Comparing base (01237e2) to head (b002b03).
Report is 53 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3022   +/-   ##
=======================================
  Coverage   96.11%   96.11%           
=======================================
  Files         383      383           
  Lines       55673    55673           
=======================================
  Hits        53511    53511           
  Misses       2162     2162           
Flag Coverage Δ
behaviourtests 3.89% <ø> (ø)
unittests 96.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@coveralls
Copy link

coveralls commented Dec 18, 2024

Pull Request Test Coverage Report for Build 12950493562

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage remained the same at 96.222%

Totals Coverage Status
Change from base Build 12905542942: 0.0%
Covered Lines: 53758
Relevant Lines: 55869

💛 - Coveralls

If I don't give its own enhancement to the masked colorised llm, either
the masked or the unmasked version doesn't find its enhancement.
@gerritholl gerritholl marked this pull request as ready for review January 7, 2025 16:20
@mraspaud
Copy link
Member

mraspaud commented Jan 8, 2025

Great idea! a couple of questions:

  • what’s happening over the baltic sea?
  • what’s happening above the arctic circle? looks like the colors are still there, but much darker. Is it the night side, and the sharp edge of the colorised part is the terminator?

@gerritholl
Copy link
Member Author

gerritholl commented Jan 8, 2025

* what’s happening over the baltic sea?

The cloud mask claims a sharp edge there between cloud-free and cloudy:

MTI1-FCI-eurol-cloud_state-20241217120000-20241217121000

* what’s happening above the arctic circle?

The cloud mask stops.

looks like the colors are still there, but much darker. Is it the night side, and the sharp edge of the colorised part is the terminator?

Maybe a resampling artefact. Redoing the resampling with gradient search, the result looks different, but still has artefacts:

MTG-I1-fci-eurol-colorized_low_level_moisture_with_vis06-20241217120000-20241217121000

Oh, gradient search is of course not great for a cloud mask.

@gerritholl
Copy link
Member Author

Resampling with nearest neighbour again, but with a larger radius of influence, the result looks better:

MTG-I1-fci-eurol-colorized_low_level_moisture_with_vis06-20241217120000-20241217121000

@gerritholl
Copy link
Member Author

With the modifier effective_solar_pathlength_corrected:

MTG-I1-fci-eurol-colorized_low_level_moisture_with_vis06-20241217120000-20241217121000

Version filled with IR 105 instead (using night_ir105 even during the day):

MTG-I1-fci-eurol-colorized_low_level_moisture_with_ir105-20241217120000-20241217121000

@mraspaud
Copy link
Member

mraspaud commented Jan 8, 2025

Nice imagery! do you have the date and time for this slot?

@gerritholl
Copy link
Member Author

do you have the date and time for this slot?

2024-12-17 12:10

@gerritholl
Copy link
Member Author

Case from today (2025-01-80 15:30), with the night part blended out using the DayNightCompositor:

202501081530-ninjo-new-low-level-moisture

@strandgren
Copy link
Collaborator

Nice, happy to see the L2 data cloud mask in action with users :)

Regarding the sharp edge over the Baltic sea, that's caused by a known bug in one of the spectral tests. A fix for this will come with the next version of the processing facility in a few weeks. As you probably know. the cloud mask is also "only" pre-operational and we are still tuning it. It seems like we are also picking up some false alarms in the Baltic sea and we also have some coastline artefacts.

Wrt. the higher viewing angles, also keep in mind that the L2 products are not generated for the full disk, but up to about 75 degrees viewing zenith angle.

@gerritholl gerritholl marked this pull request as draft January 17, 2025 13:59
For the create reference script, add the ability to use multiple
readers.
Improve behave tests.  Allow to have multiple readers.  When clipping,
use the configuration option, because reader_kwargs with multiple
readers is less practical / does not work very well.
Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

Add calibration to the fake reader channels matching 0.86 and 0.91, so
that the workaround added in 6a7a78a
does not change the total number of expected datasets in test_load.py
Use nearest neighbour when combining with cloud mask
Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

The area and the resampler were swapped for one case in the feature
definition.  Fix this.
Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

Copy link
Member Author

@gerritholl gerritholl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start behave test

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

Starting to clone and test the repository pytroll/satpy

@rymdulf
Copy link

rymdulf commented Jan 24, 2025

The testing process was executed successfully. See the test results for this pull request here!

@gerritholl gerritholl marked this pull request as ready for review January 24, 2025 11:31
Document the composites combining low level moisture with a cloud mask,
and add credit to ESSL for the colours.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants