Remove flakiness from Version linter tests

[dkijania]

Another evidence of version linter job flakiness:

Last commit on develop branch didn't generate reference version linter: da2e21f job get skipped because the changes were outside dirtyWhen condition which triggers this build

Ideas how to solve it:

We could create a separate CI job that creates the shape file for all CI runs but that could add a lot of redundant files but might be easy to implemenet? Or maybe just generate the file on-demand if it doesn’t exist? Could you create a ticket for this bug?

It might be easier to keep walking back through the branch's parent commits until we find one (up to some depth limit), but it's not very nice/elegant.

Reference files are static and only regenerated on demand when we find out discrepancy. Then we should update it in separate PR (something alike solving merges) or Run Jobs/script which will replace reference files with actual.

We regenerate reference files in nightly pipeline only. PR will only downloads what appears to be newest rerference file
We are always regenerating data in the fly and won't store them. Downloads docker/deb and run first reference dump and then actual

Crawler - we are trying to find best commit on particular branch and then download it as reference files

[dkijania]

Agreed flow with Matthew how was a reviewer of my pr:

Design include two jobs with relations

version linter depends on version linter setup

• version linter setup job:

if shapes file exists, copy it from bucket to buildkite cache
if shapes file doesn't exist, git fetch origin && git checkout origin/$UPSTREAM_BRANCH, then build and upload the shapes file and copy it to buildkite cache

version linter job:

depends on version linter setup job
by the time it runs, we know the shapes file has been uploaded, so we can confidently use it without experiencing flakes

[submarinec94]

more so for dev work improvement --- unclear where this is until he's back