Improve handling when skymap_scanner image does not exist

[tianluyuan]

Currently the latest skymap_scanner tag is used as the default version, but this does not necessarily correspond to an existing image. This could be due to failed CI builds or time delays in the sync with cvmfs. Maybe a check to see if the image exists and falling back to the latest existing image if not would be sufficient.

https://icecube-spno.slack.com/archives/C055VTS0ZTL/p1701209575163839

[tianluyuan]

@ric-evans do you know if this is still an issue? Now that we're reliant on skydiver for realtime we should ensure that should an alert come in shortly after a new tagged version of skymap scanner, before the image is up on cvmfs, everything will still run.

[ric-evans]

@tianluyuan SkyDriver uses Docker Hub as its source of truth. This 99% of the time reflects what is also in CVMFS. But, if there is a failure in the CVMFS build, there's a discrepancy. If the user does not supply a tag, or uses "latest", then SkyDriver uses the latest on Docker Hub. SkyDriver cannot access CVMFS.

I recommend always defining a tag because this can cause issues in an automated production environment.

What are your thoughts on making the tag a required field for the scan request?

[tianluyuan]

Got it, thanks for the clarification. It may not necessarily need to be a CVMFS build failure, as there is also the delay to get the image on CVMFS right? Wouldn't that also trigger the error?

I'm not opposed to making the tag a requirement, but it may mean that downstream projects will need to be updated, and also later on manually updating the tag upon each release. One idea could be to default SkyDriver to an existing tag, rather than latest, then set up an automation to update that? I'm not sure if that makes sense.. perhaps @dsschult has some ideas.