chore(batch-exports): Add batch export monitoring workflow

[rossgray]

Problem

In the past we've had some issues with events missing from certain batch exports and lack any reliable monitoring for this.

Changes

Add a new workflow in Temporal for monitoring the events exported by a given team's batch export.

This workflow is quite specific at the moment, in that it just checks 5 min batch exports (since these are the only ones using the recent_events table). It checks for a previous hour's worth of events (based on inserted_at) and stores the event count in the batch_export_runs table, so we can later reconcile this against records_completed.

I have left a few TODOs in the code for things I'm unsure about.

Does this work well for both Cloud and self-hosted?

it doesn't have an impact

How did you test this code?

Added tests

[posthog-bot]

Hey @rossgray! 👋
This pull request seems to contain no description. Please add useful context, rationale, and/or any other information that will help make sense of this change now and in the distant Mars-based future.

[tomasfarias]

Solution and implementation look really good. I have a handful of comments to address scattered all over, but nothing critical.

[rossgray]

@tomasfarias thanks for the review! Think I've addressed all your feedback. Going to spend some more time investigating what's going on with this test

[tomasfarias]

I think PR looks great. Once tests are passing this is good to merge! Good work!

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ ActivityError: Activity task failed posthog.temporal.batch_exports.monitoring in run View Issue
• ‼️ NoValidBatchExportsFoundError: Only batch exports with interval of 5 minutes are supported for monitoring at this time. posthog.temporal.batch_exports.monitoring in ge... View Issue
• ‼️ OperationalError: consuming input failed: query_wait_timeout posthog.batch_exports.service in aupdate_record... View Issue

_{Did you find this useful? React with a 👍 or 👎}