Tags get created without the CI passing

[marwaneltoukhy]

Description

Tags are done nightly, with the large test set. The issue is that these two CIs are decoupled, so if the large test set fails the tag will be created and released, which will cause issues because the docker image for that tag is not built.

For example:
tag 2023.06.23 is released but the docker image for it doesn't exist as the CI failed in the commit aeef4d0.

Proposal

Tags should only be done when we think openlane is stable, and that is when all the test sets we have pass. I think the taging CI should be the last CI to run if all the tests passed.

[donn]

The description is incorrect- but I can't disagree with the latter part.

so if the large test set fails the tag will be created and released, which will cause issues because the docker image for that tag is not built.

The tagging and Docker image building are coupled, unlike the test set.

tag 2023.06.23 is released but the docker image for it doesn't exist as the CI failed in the commit aeef4d0.

The version in question had a Docker 503 failure- Docker servers were simply experiencing an outage, so the image could not be pushed: https://github.com/The-OpenROAD-Project/OpenLane/actions/runs/5351507276/jobs/9706310682

Tags should only be done when we think openlane is stable, and that is when all the test sets we have pass. I think the taging CI should be the last CI to run if all the tests passed.

This used to be the case. We just ran into issues with that where commits were not getting Docker images because they lacked tags because the CI failed, and people who would clone master would report issues en-masse. Luckily though since then we've moved to labeling the Docker images by tag as well as by commit, so we can make it so tags specifically are only generated when the full test set passes- but we do need to create and push images for commits no matter what.