Catch panics in collector

[Kobzol]

The way the site endpoint is currently set up, it expects that each collector bench_next invocation that starts a collection will eventually call the /perf/onpush endpoint. However, if the collector crashes (which happened today because of a DB timeout), it won't ever call the endpoint. This will cause the following successful collection to mark both collections as completed at once when it calls /perf/onpush, which is not intuitive and causes weird situations with GH comments (two comments being posted at once). We should rather just eagerly notify about the error.

We could perhaps somehow store a special error into the DB, letting it know that a panic has happened, but since the panic can happen because of DB not behaving reliably.. it's hard to say if it would work.

[Mark-Simulacrum]

I'm fine with this but I wonder if we should just poke on push within the wrapper script? Or in addition? That seems more reliable to me than catching panics, and should be equally effective?

[Kobzol]

We would need to somehow communicate to the bash script that a benchmark took place though (otherwise it shouldn't call onpush). Or make bench_next blocking, but that would not keep the collector in sync with git pushes.

[Kobzol]

onpush can't be triggered more than once per a minute, so it would probably be harmless to call it a second time from the bash script, although it's quite hacky. If it didn't make the second call in the 60 second interval, it could in theory finish an ongoing perf. run, but there should not be any ongoing perf. run, since the script won't start a new one until onpush is called..

This means that it should be ok to just call the endpoint always, from the script. We will just spam the website with a request every two minutes, but it should work.

[Mark-Simulacrum]

Happy to merge as-is or with the every-2-minutes-when-idle hit.

[Kobzol]

I thought about it a bit more and I don't like the idle call, because calling onpush will basically clear the site's cache (both the DB index and the landing page). This would mean that if there's no collection going on, the site would be needlessly refreshing its cache all the time, that doesn't seem good.

So let's try the panic catching approach.