Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes sofar notifications #1205

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

Yannick-Rousseau
Copy link
Contributor

@Yannick-Rousseau Yannick-Rousseau commented Jul 25, 2024

Ok, so I made a few changes in the sofar_nrt files.
Note, the Aqualink token can be ignored, as it should be already implemented in a previous merge

  1. adding a try/except in the main > that allows to skip spotters when something is wrong.

  2. adding a series of conditions in the process_wave_source_id > they allow to skip if the spotters are not in the right coordinates, if deployment date missing, api token missing, spotter ids or locations are duplicated (to remove old deployments), ...
    The main thing with that is coordinates. The code checks for outliers (coordinates on api within 0.1-0.02 deg of metadata) and errors (>0.1 deg). The values can be changed.
    If there is an outlier (no matter the length) in the data but the spotter is back in position, it downloads the data and warns about it.
    If there is an outlier at the end of the data, the data is downloaded up till the first outlier (and coded to start again once the spotter is back in place).
    If there is an error in coordinates, if it is a temporary one (<7 days), it considers that the spotter was moved to shore and they forgot to switch it off, no problem.
    If the last data point is an error, it warns that it is so, and download up until the error.
    If the error in coordinates persists more than a week, it warns to check the coordinates with the facility.
    (The choice of 1 week was due to a few spotters that were moved for 5 days, then placed back).
    Every time, the data up until the outlier or error is downloaded, and will restart at a later date if it resolved, or warn to change something in metadata if needed.

  3. messages. Not only the errors and warnings are logged in the log for each spotter, it also collects them all and give a summary at the end of the log, which looks something like that:


2024-07-25 12:27:59.879313 :
Partial data download. The following buoys were skipped due to errors:
'SPOT-0411 ( Collaroy )': 'The location name is duplicated and data was not downloaded for either. Please remove old deployment from metadata.',
'SPOT-1462 ( Apollo Bay )': 'Last data point (2024-06-13 10:05:31) is an outlier (<10km). Data until 2024-06-13 07:05:31 has been downloaded, but download for later data, will only resume when buoy is back at location.',
'SPOT-1542 ( Mermaid Reef )': 'API error: missing token. Please note that institution names are case sensitive.'

The following warnings were raised:
'SPOT-1632 ( Lakes Entrance )': 'Deployment start date missing. Download of data skipped.',
'SPOT-0312 ( Boambee )': 'No data available BETWEEN 2024-01-01 00:00:00+00:00 AND 2024-02-01 00:00:00+00:00.',

  • Note that 'deployment start date missing' usually denotes a buoy IMOS does not handle anymore but could indicate missing metadata.

  1. finally the notification.
    Devopps decided against setting up a notification system for this, therefore I went with logging in a summary message at the end of each run. If at a later date someone wants to send that message as a notification (slack, email or other), it can be (but needs to be discussed with them on how to do that in prod).

  2. oh yeah, and unitest for duplicates s[potter ids. That requires the metadata in the test folder to be the one used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant